The underlying connection was closed: The connection was closed unexpectedly.

This had to be one of the most vague, non-descriptive errors I have received in my entire career as a computer programmer. Not to say that it was the worst message, there are still a few that are better than this one, but this is definitely up there in the list.

I wrote some ASP.Net server-side code that verified a user's entry of a website address. It did this by creating an HTTPWebRequest and retrieving the webpage data for the URL that the user entered. If the code retrieved HTML data, then the website address supplied was correct, otherwise the website URL supplied is considered invalid. The idea is sound, and in most cases worked perfectly...that is until today. I get a support ticket from a client that stated they were receiving an error that said: The underlying connection was closed: The connection was closed unexpectedly.

I immediately tried to reproduce the error, because as we all know end-users cannot always be trusted to tell the whole truth. However, to my surprise this one was a valid complaint. So I started scouring the code as well as the internet to try and find the cause and remedy for this poorly worded error. I found others with similar problems, some claimed it was a timeout issue, others claimed it was an encoding issue, and other claimed it was a keep-alive issue. Alas, all those remedies did nothing to fix my problem. I spent the better part of the morning trying every trick I knew to determine the problem. I even began picking apart the return HTTP Headers using Fiddler.

Ultimately I determined that the reason the connection was being closed, was because the web server was forcibly closing it. It had nothing to do with me having bad code...or not entirely at least. The web server was rejecting any HTTP connection requests that did not also identify the platform that the request was coming from. An annoying feature to say the least. So I decided to "spoof" a MSIE 8.0 platform running on Win7. Now that my code was identifying itself as a particular platform, the web server allows the HTTP request through and the HTML data was finally retrieved.

I determined the UserAgent string by using a free online generator: http://user-agent-string.info

I have the code shown below. I hope that this post can help someone out there having a similar issue.

	Public Shared Function GetWebPage(ByVal p_strURL As String) As String
	'***************************************
	' Initialize Variables
	'***************************************
	Dim objWebRequest As HttpWebRequest
	Dim objWebResponse As HttpWebResponse = Nothing
	Dim objStreamReader As StreamReader
	Dim strWebPageData As String = ""
	'***************************************
	' Create WebRequest
	'***************************************
	objWebRequest = WebRequest.Create(p_strURL)
	objWebRequest.Method = "GET"
	objWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)" 'MSIE 8.0 (Windows 7)
	'***************************************
	' Read Returned Data
	'***************************************
	objWebResponse = objWebRequest.GetResponse()
	If (objWebResponse IsNot Nothing) Then
	objStreamReader = New StreamReader(objWebResponse.GetResponseStream)
	strWebPageData = objStreamReader.ReadToEnd.Trim()
	objStreamReader.Close()
	End If
	'***************************************
	' Return WebPage Data
	'***************************************
	Return strWebPageData
	End Function

view raw GetWebPage.vb hosted with ❤ by GitHub