More efficient way to write to an output file in Powershell

advertisements

I have a Powershell script i'm using to parse each row in a file, reformat it, and write the new string to an output file. It works fine with an input file with a few hundred lines. However, I need to ultimately run it against a file with a few million lines, and I have been waiting hours and it still hasn't finished. Following this post, I think I need to put Write-Output outside of the loop, but i've been unsuccessful so far.

This is my current code:

Foreach ($line in Get-Content $logFile) {

    $arr = $line.Split()

    $port1 = $arr[9].Split(":")

    $port2 = $arr[11].Split(":")

    $connstring = '|' + $port1[0] + "|" + $port1[1] + "|" + $port2[0] + "|" + $port2[1] + "|" + $arr[4] + "|"

    Write-Output $connstring | Out-File "C:\logging\output\logout.txt" -Append
}

An example of an input string is:

06/14-04:40:11.371923  [**] [1:4:0] other [**] [Priority: 0] {TCP} 67.202.196.92:80 -> 192.168.1.105:55043

And I need to reformat it to this:

|67.202.196.92|80|192.168.1.105|55043|other|

Any help is much appreciated!


If you use a -ReadCount on the Get-Content it will have the effect of streaming the file one row at a time rather than having to read the entire file in to memory. I suspect that moving write operation outside of your loop might be faster. Less variables and steps inside your loop will probably help too.

Assuming the fourth element after the split doesn't contain a colon (you didn't supply and example of your file) then something like this is should do the trick:

Get-Content $logFile -ReadCount 1 | % {
    '|' + (($_.Split()[9, 11, 4] -replace ':', '|') -join '|') + '|'
} | Out-File "C:\logging\output\logout.txt"