Registry can only be edit with certain tools, usually it isn't cleaned up after an application is un-installed, if you move your app to another machine then it's hard to get registry settings to move as well.
The little sample attached should give you an idea:
Note the name of the file under VS is called app.config, however at runtime the file needs to be called (yourappname.exe.config - VS takes care of this for you).
Build the sample and go to the bin directory generated and run the exe, click the button and see what the label is set to.
Edit the .config file in any text editor and click the button again (remembering to save the file!) appconfig.zip