Today I encountered a strange error during the configuration of a Build Controller in TFS. We installed and configured the first Build Controller for a TFS Instance, everything went good, but both controllers and agent are marked with stopped icon, even if status is “ready”

image

Figure 1: Controller and agents are marked as stopped even if they are in Ready State

I immediately looked into Event Viewer, but absolutely no clue of what is happening. I tried creating and scheduling a build, but it starts, then remains silent forever. The build system was not working. I remember a post by Richard where he had the same problem, but I’m not in that scenario. I checked DNS, tried to ping the server and everything is ok, but builds never starts and there are absolutely no error in event viewer.

Then I noticed that in the upper section of the Build Server there is another link called Details… that usually is not there. If I clicked on that link it told me that the controller is not able to communicate with TFS because he got a 500 internal error response.

This is extremely painful, because it means that something in the Application Tier is not working properly, so I immediately remote desktop into the TFS machine and looked at the Event Viewer of the server. This time the error is there and luckily enough it was simple to fix.

System.ServiceModel.ServiceHostingEnvironment+HostingManager/42931033
 Exception: System.ServiceModel.ServiceActivationException: The service '/tfs/queue/test/Services/v4.0/MessageQueueService2.svc' cannot be activated due to an exception during compilation.  The exception message is: Memory gates checking failed because the free memory (176160768 bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element.. ---> System.InsufficientMemoryException: Memory gates checking failed because the free memory (176160768 bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element.
   at System.ServiceModel.Activation.ServiceMemoryGates.Check(Int32 minFreeMemoryPercentage, Boolean throwOnLowMemory, UInt64& availableMemoryBytes)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CheckMemoryCloseIdleServices(EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
   --- End of inner exception stack trace ---
   at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
   at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath, EventTraceActivity eventTraceActivity)
 Process Name: w3wp
 Process ID: 2768

This is a typical error you can encounter if you install TFS in a single machine configuration. If you follow general guidance on MSDN the single server approach is ok for groups up to 500 users, with 4 GB of ram and 1 disk at 10k. Single server maintenance is easier and for small team is probably the best configuration, but you need to be aware of one possible problem: SQL Server is greed about memory.

The problem is that SQL Server tends to use all available memory, until the system starts becoming really, really slow because it has no free memory for other processes. Whenever you install TFS in a single machine environment, is a good suggestion to limit maximum amount of memory available to SQL Server, leaving space for the AT to work properly. I have no gold number to give you, but if you have a single machine with 4 GB of RAM, usually I limit SQL Server to a maximum of 2 GB. In this specific situation I remember talking about this configuration, but it was never done; this results in SQL Server using about 3 GB of RAM in a 4 GB machine, leaving no space for WCF Service to starts.

Lesson learned: Whenever something goes wrong in TFS, always have a look at events viewer of all machines involved in the process (AT, SQL, Build, etc) because root error could originates in another machine and not in the one you are looking at. As a rule of thumb, if something went wrong, always look at the AT machine Event Viewer.

Gian Maria.

Tags:

No comments

During a build you can ask MsBuild to deploy on build using the switch /p:DeployOnBuild=true as I described in previous posts. This is mainly used to deploy the site on IIS thanks to WebDeploy, but you can also use WebDeploy to deploy on a disk path. The problem is that the path is stored in publication settings file, but what about changing that path during a Build?

The answer is simple, you can use the /p:publishUrl=xxx to override what is specified inside the publication file and choose a different directory for deploy. Es.

msbuild WebApplication1.csproj /p:Deploy
OnBuild=true /p:PublishProfile=Profile1 /p:publishUrl=c:\temp\waptest

Thanks to this simple trick you can instruct MsBuild to store deployed site in any folder of the build server.

Gian Maria.

Tags:

No comments

Some time ago a friend asked me the easiest way to get code from a specific folder and a specific version in TFVC. The goal is avoiding using Get Specific Version because he do not want to overwrite the Workspace folder he is using, he want also to avoid creating another workspace only to do a one-shot get of a folder.

It turns out that the easiest way to accomplish this task is from Web Interface, because it has the capability of browsing and downloading code as zip. You can simply navigate to the CODE hub in web interface, choose the folder you want to download and use the context menu of desired folder to download everything as a single zip file.

image

Figure 1: Browse and download code from the Web Interface.

But wait, this will download the latest version of the code, not a specific version. The cool part of the web interface is that if you just append #version=xxx where xxx is the changeset-id you want to download, you can browse the code of that specific changeset, and you can also download as Zip that specific version. If you just look at the url, you can easily spot out that downloading the code as zip is just a matter of calling the right url

https://gianmariaricci.visualstudio.com/DefaultCollection/Experiments/_api/_versioncontrol/itemContentZipped?repositoryId=&path=%24%2FExperiments%2Ftrunk&version=96&__v=5

You can simply change the version parameter and you are able to download every version of your code as zip with a simple call and without resorting to API or external tool. Just copy and paste url inside your browser and you are done. If you have not previously authenticated with your TFS or VSO you will be prompted for credentials, then the file is downloaded.

I’ve showed you examples with Visual Studio Online, but you can use the very same technique against your on-premise TFS.

Gian Maria.

Tags:

Comments Off

It could happens when you clone a Git Repository with submodules, issuing a git submodule update command, you are prompted with this error error.

Cloning into ‘src/xxxx’…
Warning: Permanently added the RSA host key for IP address xxx.xxx.xxx.xxx to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

If you search in the internet for the cause of errors, you can find some people suggesting that the url specified in .gitmodules file is wrong and should be changed, here is my .gitmodule

[submodule "src/CQRS"]
     path = src/CQRS
     url = git@github.com:xxxxxx/cqrs.git
     branch = master
    

You could change the url configuration to https url and everything works, but this is not the perfect solution, because the address git@github.com is perfectly valid, but probably there is some problem with your RSA keys stored in Github (or you never configured RSA Keys for your account). In my situation, my RSA Keys had some problem and I needed to recreate another one. If you do not know what a RSA key is and how to create a RSA Key to connect to github I strongly suggest you reading the guide: Generating SSH Keys.

Once you configure a valid certificate in github your submodule should word without problem.

Gian Maria.

Tags: ,

Comments Off

Using DPAPI to encrypt password in builds suffers from a serious drawback, the password can be decrypted only by code that runs on the very same computer used to encrypt the password. You can overcome this limitation using roaming profiles, but it is not a option in many scenario. Another technique to enable multiple build servers to decrypt a password is using certificates. In PowerShell is really easy to use a certificate to encrypt/decrypt string and this article will show you how to secure a password in TFS Build definition using Certificates.

First step is generating a valid certificate, this is easy thanks to MakeCert utility, just open a Developer Command Prompt and type

MakeCert.exe -r -pe -n “CN=www.cyberpunk.local” -sky exchange -ss my -len 2048 -e 01/01/2020

This command creates a certificate in your certificate store. To view it you should type certmgr to open the Certificates Manager Console. Your newly created certificate should now appear in the list of certificates.

image

Figure 1: Your newly created certificate appears in certificate store.

Double Clicking the certificate opens detail windows where you can find the value of Thumbprint property (Figure 2). This property is useful because it uniquely identifies this certificate in certificate store, and it should be used to load correct certificate in PowerShell script.

image

Figure 2: Grab the Thumbprint of the certificate in detail pane.

You should grab the thumbprint from the certificate details, it should be a string like: f02e4c0e13e26d25065cc0db1d03450acaef90d6. Just copy from the details pane, remove all the spaces and you are ready to use this certificate to Encrypt and Decrpyt a string. Here is the PowerShell script that uses this certificate to Encrypt and then Decrypt a string.

$cert = Get-Item -Path Cert:\CurrentUser\My\f02e4c0e13e26d25065cc0db1d03450acaef90d6 -ErrorAction Stop
                                            
$decrypted = ""
$bytesDecrypted = ""
$bytesToDecrypt = ""
$pwd = 'mySecurePwd'
$enc = [system.Text.Encoding]::UTF8
$pwdBytes = $enc.GetBytes($pwd) 

$encryptedPwdBytes = $cert.PublicKey.Key.Encrypt($pwdBytes, $true)
$EncryptedPwd = [System.Convert]::ToBase64String($encryptedPwdBytes)

Write-Output "`r`nEncrypted Password IS:"
Write-Output $EncryptedPwd

#Now decrypt.
$bytesToDecrypt = [System.Convert]::FromBase64String($EncryptedPwd)
$bytesDecrypted = $cert.PrivateKey.Decrypt($bytesToDecrypt, $true)

$decrypted = $enc.GetString($bytesDecrypted)

Write-Output "`r`nDecryptedPassword is: $decrypted"

As you can see using the certificate is straightforward, you just grab a reference to certificate with the Get-Item using the thumbprint and then it is just a matter of calling Encrypt method of the public key object stored inside the certificate and convert to Base64 to be conveniently represented as a string. To decrypt you can call Decrypt method of the Private Key object.

If you does not know how a RSA key pair works, here is the basic concept. The certificate contains two keys, one is the Public Key, and the other is the Private Key. You use the public key to encrypt a string and the resulting encrypted string can be decrypted only by the Private Key. Public key used to encrypt the string cannot be used to decrypt it. This is the reason why RSA is called an Asymmetric Algorithm.

Now return to the Certificate manager, and press “Export” button to export the certificate, choose to export only the public key, choose DER format and export the certificate to a folder of your computer. Then press “Export” again but now choose to export the private key, choose the PKCS format and then use a password to protect the exported file. A password is required because the private key is the information that you need to keep secure from the eyes of the public. Now you should have two exported files, one with .cer extension that contains only the public key, the other with .pfx extension containing also the private key.

image

Figure 3: Exported certificates in my filesystem.

Now remove the certificate from Certificate Manager, and try to run again the script to encrypt and decrypt the string, you should now receive an error from Get-Item, telling you that the required path does not exists. This confirms that the certificate was removed from your certificate store. Now press the “import” button on Certificate Manager and import the previously exported public key, the file public.cer in my example and run the script again to verify that you are able to encrypt a string but you cannot decrypt, because you have only the public key.

image

Figure 4: Now that you have only the public key, you are able only to encrypt and you could not decrypt anymore the string.

To verify that you can decrypt password using Private Key, press Import button again on the Certificate Store and import the Private.pfx certificate. You will be prompted to enter the password you use during the export, and if you look at the options you can verify that there is an important option called “mark this key as exportable”, that is turned off by default.

image

Figure 5: Importing a certificate with Private key in certificate store

This option is really important, because this prevents users to further export the private key to another computer. Once you imported the Private certificate, you can verify that pressing the Export button does not permit you to choose the option to export the Private Key. Once the certificate with private key is imported you are now able to decrypt the password.

Thanks to asymmetric key you can simply store the certificate with the public key in any network share, everyone can access the certificate, install it and securely encrypt a password because only people with the private key can decrypt it. Now you should login in every Build Machine with credential of TfsBuild, and import the certificate with the Private key to personal certificate store, do not mark the key as exportable. Now you can use Certificates instead of DPAPI to encrypt and decrypt password during the build. This script will work in every build server where you have installed the certificate for the user TfsBuidl. Here is the script.

Param
(
[string] $url = "http://webtest1.cyberpunk.local:10000/MyService/Index",
[string] $username = "",
[string] $password = ""
)

Write-Host $password
$cert = Get-Item -Path Cert:\CurrentUser\My\f02e4c0e13e26d25065cc0db1d03450acaef90d6 -ErrorAction Stop
Write-Host $cert.PrivateKey.KeySize
$bytesToDecrypt = [System.Convert]::FromBase64String($password)

$bytesDecrypted = $cert.PrivateKey.Decrypt($bytesToDecrypt, $true)
$enc = [system.Text.Encoding]::UTF8
$plainText = $enc.GetString($bytesDecrypted)

Write-Host "Invoking-Service"

$retValue = Invoke-RestMethod $url"?username=$username&password=$plainText"  

Write-Host "ReturnValueIs: "$retValue.Message

This script simply decrypt the password and call a simple REST service just to verify that the password was decrypted with success. Here is the output of the build.

image

I want to remember again that this technique does not secure the password for people that can schedule a new build in TFS. Any person that can schedule a new build can create a PowerShell script that decrypt a password and print it in build output, grab an encrypted password from another build definition and let this script decrypt it.

This is the same technique I used with the above script where I dump the password in clear format in Build output, just to verify that decryption is ok. Any other user with permission to schedule a build could have scheduled a build running a similar script to decrypt password store by other people.

Gian Maria.

Related Articles

Tags:

1 Comment