Monday, January 13, 2014

Transferring AWS S3 Buckets

tl;dr - Install awscli (pip install awscli), then:


aws s3 sync s3://srv-salt .   --profile myFirstProfile
aws s3 mb srv-salt --profile mySecondProfile
aws s3 sync . s3://srv-salt --profile mySecondProfile


Install awscli
A quick Google search sent me to this StackOverflow answer, which made awscli news to me. I went with installing via Pip so:


apt-get install python-pip
pip install awscli

Configure awscli
I have a few profiles to manage, so, following the docs, I configured each of them with:

aws configure --profile myFirstProfile
aws configure --profile mySecondProfile

Transfer the Buckets
Once I tested out each profile with a describe-instances, copying an entire bucket to another region was simple:

aws s3 sync s3://srv-salt .   --profile myFirstProfile
aws s3 mb srv-salt --profile mySecondProfile
aws s3 sync . s3://srv-salt --profile mySecondProfile

And that was it! I love it when things are easy...

Sunday, January 5, 2014

Salt states for salt-api

If you read my previous post and would like to play the home version, I've written a few states to turn a Salt minion into a (insecure, testbed) install of salt-api.

You can find them over here on Github.

Share and Enjoy!

Thursday, January 2, 2014

Salt-API, A Crash Course

The salt-api docs do a great job but it's a bit unclear on what it takes to get a barebones salt-api proof of concept up and running. No official deployment, no top cover from a web server. Just a Salt master with salt-api running on top of it. We'll cover that here.

I (heavily) referenced the rest_cherrypy section of the docs to put this together.

This post assumes that you already have a Salt master and at least one Minion connected to it.

Install salt-api
The docs seem to mention that Pip is the best method. Also, we're going the CherryPy route so:

pip install salt-api cherrypy

Configure CherryPy
Very simple. In your master config file:
 
rest_cherrypy:
  port: 8000
  ssl_crt: /etc/pki/tls/certs/localhost.crt
  ssl_key: /etc/pki/tls/certs/localhost.key

Notice the SSL lines. We can create a self-signed keypair very easily with Salt:

salt-call tls.create_self_signed_cert


Configure External Auth
Salt-api leans on the eauth system for authentication. For our quick and dirty, we can just allow our user to do everything. Back in the master config file:
external_auth:
  pam:
    saltuser:
      - .*
Turn On the Awesome
We're done configuring! With the Salt master & minion running, start salt-api:
salt-api 

You can optionally pass salt-api the -d option to put it into daemon mode.

With salt-api runnig. Test it out:
curl -k https://localhost:8000

{"status": "401 Unauthorized", "return": "Please log in"}



Success!

We'll need to log in. That looks like this:
 curl -ksi https://localhost:8000/login \
> -H "Accept: application/json" \
> -d username='saltuser' \
> -d password='password' \
> -d eauth='pam'

{"return": [{"perms": [".*"], "start": 1388722947.535586,
"token": "f72309d0ee425193bc8b763a0092470bbabab6bc",
"expire": 1388766147.535588,
"user": "saltuser",
"eauth": "pam"}]}


This gives us a token. In this example the token is  f72309d0ee425193bc8b763a0092470bbabab6bc. Let's use it to run a test.ping.

 curl -ksi https://localhost:8000 \
-H "Accept: application/x-yaml" \
-H "X-Auth-Token: f72309d0ee425193bc8b763a0092470bbabab6bc" \
-d client='local' \
-d tgt='*' \
-d fun='test.ping'

HTTP/1.1 200 OK
Content-Length: 25
Vary: Accept-Encoding
Server: CherryPy/3.2.4
Allow: GET, HEAD, POST
Cache-Control: private
Date: Fri, 03 Jan 2014 04:31:13 GMT
Access-Control-Allow-Origin: *
Content-Type: application/x-yaml
Set-Cookie: session_id=f72309d0ee425193bc8b763a0092470bbabab6bc; expires=Fri, 03 Jan 2014 14:31:13 GMT; Path=/

return:
- salt-api: true




Not bad. Let's try making it actually do something.
curl -ksi https://localhost:8000 \
-H "Accept: application/x-yaml" \
-H "X-Auth-Token: f72309d0ee425193bc8b763a0092470bbabab6bc" \
-d client='local' \
-d tgt='*' \
-d fun='cmd.run' \
-d arg='echo "hi salt-api!"'

HTTP/1.1 200 OK
Content-Length: 33
Vary: Accept-Encoding
Server: CherryPy/3.2.4
Allow: GET, HEAD, POST
Cache-Control: private
Date: Fri, 03 Jan 2014 04:34:19 GMT
Access-Control-Allow-Origin: *
Content-Type: application/x-yaml
Set-Cookie: session_id=f72309d0ee425193bc8b763a0092470bbabab6bc; expires=Fri, 03 Jan 2014 14:34:19 GMT; Path=/

return:
- salt-api: hi salt-api!



Neat!

You can see where we can go from here. Once you've got a RESTful interface to your Salt master, your creativity is the only limit from what you can do from there.

Sunday, December 15, 2013

The Industrial Model Isn't Dead - It's Migrating

People love something 'made by hand'. Tell them that, to make your table, you collected driftwood from the beach. Or your vanilla extract is different because you've met the farmers and you get your alcohol for extraction from the (craft) distillery in your town. A beer made with intent will always taste better than a beer made for a bottom line. Tell someone a story, or how you painstakingly made each groove, and people listen. 

Software is not like that. Systems, deployments, IT, and data centers are not like that.

No one (okay, very few people) cares about the love that was put into some software. They're not concerned with the initial itch that needed scratching. They don't look forward to hearing about how the author put countless hours of effort into automating some mundane routine so that the user wouldn't even notice that it had been done for them. 

No one builds data centers by hand now and no one wants to hear about how someone built a data center by hand (rather - the people who do still build them by hand soon won't; either because they'll have upgraded their job or been left behind).

This leads to an interesting contrast between hand made goods and software. We like to hear that a coffee table took two weeks of carving and staining. But a data center? We want to know that a system can be deployed 'with the push of a button'. Never mind made with care, we want the actual creation of a system to be as automated and hands free as possible. Where goods are moving away from the bulk, industrial manufacturing model, IT systems and application deployment is running straight into it. More systems, faster, for less work (and, thus, cost).

The industrial model is still very much alive. We're just moving it from farms and goods over to data centers.

Tuesday, November 26, 2013

Getting Religion - Salt

Apparently I was on a quest and didn't even know it. Automation is great (essential, even), but at some point you need to wrap your arms around the whole thing. After spending some time with Puppet (and deciding it wasn't for me), I found Chef to be pretty approachable. I like the native versioning of recipes and the hands-free method of creating workstation separate from the master (the fewer hands that touch the master, the better). But there was a definite learning curve to be overcome.

Then I found Salt.

Pure simplicity.

How do I install Salt on CentOS?
yum install salt-master

How do I use Salt to install Apache? 
salt myMinion pkg.install httpd

What about all my webservers? 
salt -N myWebServers pkg.install httpd

Install Apache, pull down the latest version of my code, and restart Apache if the code changes!

my-awesome-app:
  pkg.install:
    - name: httpd
  git.latest:
    - name: http://github.com/awesome/sauce.git
    - target: /var/www/html/awesomesauce
  service:
    - name: httpd
    - enable: True
    - running
    - watch:
      - git: my-awesome-app

And done! Every server. Latest code. Apache keeps up. All in easy to read yaml. 

So simple.

Hell, it even works with Powershell. Fully automated MSSQL installation? Done. Local user management? Easy. I'm confident that, with the proper Salt setup, you can do away with System Center Config Manager. The implications are huge for people willing and able to do it themselves (instead of paying for Microsoft licenses and consultants).

I'm hoping to make some big changes in my environment with Salt. The more work we can get the machine to do, the more time we have to work on more interesting problems. 

Monday, March 4, 2013

Chef 11 Deployment - Lessons Learned

I had rolled out a few proof-of-concept deployments of Chef server on CentOS and Ubuntu, each complete with their own workstation. I thought I had it down.

I was wrong.

This post will serve as a reference for my future self. Hopefully you, dear reader, have landed here after Googling an error. I hope I can save you some misery. This rollout was Chef Server 11.0.6 on Ubuntu.

Changing The Hostname

I was rolling out my deployment on Amazon AWS and didn't particularly care for the default machine names of an IPv6 address. I changed the hostname of my Chef server before doing anything, figuring it would affect nothing. I was wrong.

As it turns out, your machine's hostname must resolve successfully in order for chef-server-ctl reconfigure to complete successfully. My fix was exactly as described in that Opscode ticket - drop a line in /etc/hosts:

<floating IP> ubuntu-chef-11-server

Onward...

Your Workstation and Creating Your Own knife.rb

The latest docs on setting up your workstation mention knife.rb seven times but the page assumes that you're using Hosted Chef or Private Chef. Not your own Chef server. As such, I consulted some older (but still very useful) docs to create my own knife.rb. The doc says that the API listener sits on port 4000 and the Web Interface on 4040. This is not the case.Both listen on port 80. I lost two hours looking at this error:

[root@chef-11-workstation chef-repo]# knife user list
ERROR: Connection refused connecting to ubuntu-chef-11-server:4000 for /users, retry 1/5
ERROR: Connection refused connecting to ubuntu-chef-11-server:4000 for /users, retry 2/5
ERROR: Connection refused connecting to ubuntu-chef-11-server:4000 for /users, retry 3/5
ERROR: Connection refused connecting to ubuntu-chef-11-server:4000 for /users, retry 4/5
ERROR: Connection refused connecting to ubuntu-chef-11-server:4000 for /users, retry 5/5

This is exacerbated by the default value provided when running knife configure:
Please enter the chef server URL: [http://localhost:4000] 

Makes you think port 4000, right? Wrong. It's port 80.

In Summary

  1. If you must change the hostname, confirm that the new name can be resolved (via /etc/hosts, for example). 
  2. The API listener and the WebUI both listen on port 80.

Wednesday, February 6, 2013

Sustainable Powershell - Cmdlets

Since we're automating work, and (currently) a lot of the work involves Windows ecosystems, the natural tool of choice is Powershell. Let's look at some general guidelines for writing sustainable Powershell cmdlets.

For the purpose of this post, I'll use functions and cmdlets interchangeably as they can both be considered atomic work units.

Powershell Cmdlet Guidelines

Cmdlets shall:

Use the Verb-Noun Naming Scheme

Using this scheme forces you to think more about what your functions/cmdlets should be doing. It's the decided 'best practice' from Microsoft and when you import a module that violates this scheme, you get a nice nagging message from Powershell telling you exactly that.

Protip: Not that you should use it, but you can pass the -DisableNameChecking parameter to Import-Module to suppress the warning. You should know, however, that if you need to suppress the warning, you're probably developing your cmdlets the wrong way.

Use Camel Case for Capitalization

For instance, Get-DetailsFromReallyLongName.

The corollary to this is use descriptive names. The year is 2013. We have plenty of disk to store code with long names. Use it.

Cmdlets Do Not Hold State

 Cmdlets do stuff, they do not hold things. If you're familiar with RESTful interfaces and/or stateless architecture, move to the next point because you already understand.

For the rest of us: A cmdlet should not hold onto a value. That is, it should not store something in a global/environment variable. It can check some environment variables if it needs to make a decision, but it should not communicate with the user/other cmdlets via manipulating those variables.

If you need data, put it in/get it from a data source, such as a database, the Windows registry, or even a flat text file. Don't store it in memory (e.g. a variable) because you'll create problems for yourself that distract you from the real work.

Use "Approved" Verbs

Personally, I'm a little bitter that the creators of Powershell drew a line around what we can and cannot use when writing our own modules but that doesn't change the fact that we have to deal with it. And besides, it does help to keep you in the right mindset when writing functions/cmdlets.

When writing modules, your functions need to start with 'approved' verbs. To get a list of those verbs, use the command Get-Verb or see Microsoft's writeup.

Do One Thing. Do It Well.

See also the Unix Philosophy of software development. Forty years of tireless development and crushing testing environments have made this philosophy even more relevant.

Use Command-Line Parameters

When thinking of how to pass information to your cmdlet, think in the same way you write your cmdlets/functions: simple and atomic. Don't pass an object representing information to your cmdlet (because that breeds complexity, which leaves you prone to errors). Instead, simply pass the information.

For example, if a cmdlet needs information about a server, don't create an object just to represent that server. Instead, have your cmdlet accept the text version of that information.

Don't Do This
Create-NewServer -serverDetails complicatedObject 
Where complicatedObject is an object or a hash with values describing the server.

Instead, Do This
Create-NewServer -ServerName worker01 -IP 192.168.10.1 -Netmask 255.255.255.0

Writing your function will take more time and the incantations will be more verbose. B-U-T in the future when (not 'if', but 'when') you need to update that function, making the required changes will be so much easier than updating code that expects more complex input.

In summary, this is the state we're in today. The environment will change and constraints will improve but, in the meanwhile, following these guidelines will reduce (not eliminate!) tomorrow's maintenance cost of the software you write today.