Managing Services on Linux with systemd

 You've read all about systemd, the new Linux init daemon. You know what it does, and why. Now it's time to dig in and learn how to make it sit up and beg — or at least start, stop, and get information on services.

Starting and Stopping Services

My earlier piece, "Here We Go Again, Another Linux Init: Intro to systemd" discusses the concepts behind systemd and what it is supposed to do. Now it's time to learn how to use it to control services on our systems. systemd is backwards-compatible with sysvinit and Upstart, so you can try it out by installing it on any Linux that uses sysvinit or Upstart without a lot of extra work. Arch Linux, Debian, and OpenSUSE all include systemd in their software repositories.

A conspicuous omission from distros that support systemd is Ubuntu. There are various reasons given for this, and I don't care because I'm tired of geekfights and just want to get on with things. Another way is to fetch yourself a copy of Fedora 15 or 16, which runs systemd by default.

Figure 1: systemadm for graphical systemd administration.

systemadm is a nice graphical systemd manager (figure 1). It's still a baby so it might crash or something, but you can try it by installing the systemd-gtk package on Fedora, the systemd package on Arch, or the systemd-gui for Debian. No, of course distros cannot use consistent package naming, because that is against the rules.

As pretty as systemadm is, let us adjourn to the command line for the rest of this article. Watch the prompts in the examples to see which ones require root privileges. You can see the status of everything systemd controls by running systemctl with no options:

$ systemctl
UNIT                                              LOAD   ACTIVE SUB       JOB DESCRIPTION
proc-sys-fs-binfmt_misc.automount                 loaded active waiting       Arbitrary Executable File Formats File System Aut
sys-devices-pci0...000:00:1b.0-sound-card0.device loaded active plugged       82801I (ICH9 Family) HD Audio Controller
sys-devices-pci0...-0000:05:00.0-net-wlan0.device loaded active plugged       Centrino Wireless-N 1000

How do you see only available services, running or not?

$ systemctl list-units -t service --all
UNIT              LOAD   ACTIVE   SUB   JOB   DESCRIPTION
ceph.service      loaded inactive dead        LSB: Start Ceph distributed filesystem daemons at boot time
chronyd.service   loaded active   running     NTP client/server
cman.service      error  inactive dead        cman.service

These will spit out a whole lot of output, probably a hundred lines or more. Want to see only active services?

$ systemctl list-units -t service

You can check the status of any individual service, such as the cman.service, which has an error flag in the previous example:

$ systemctl status cman.service
cman.service
   Loaded: error (Reason: No such file or directory)
   Active: inactive (dead)

How nice, it tells us what the problem is. Here is what a normal running service looks like, with complete information on the service:

$ systemctl status sshd.service
sshd.service - OpenSSH server daemon
   Loaded: loaded (/lib/systemd/system/sshd.service; enabled)
   Active: active (running) since Thu, 15 Dec 2011 12:11:05 -0800; 2h 26min ago
 Main PID: 2091 (sshd)
   CGroup: name=systemd:/system/sshd.service
      \   2091 /usr/sbin/sshd -D

On Fedora you can still use the old service and chkconfig commands. But why not start learning the new systemd commands? This is how to start a service:

# systemctl start sshd.service

Or use restart to restart a running service. This stops a service:

# systemctl stop sshd.service

Those are in effect only for the current session, so if you want a service to start at boot do this:

# systemctl enable sshd.service

And that's all. No hassling with startup scripts. This disables it from starting at boot:

# systemctl disable sshd.service

You can check to see if a service is already running; 0 means it is currently running and 1 means it is not:

$ systemctl is-enabled sshd.service; echo $? 
enabled
0

You can use systemctl halt, poweroff, or reboot to shutdown or reboot the system. All three send a wall message to users warning that the system is going down.

Processes, cgroups, and Killing

systemd organizes processes with cgroups, and you can see this with the ps command, which has been updated to show cgroups. Run this command to see which service owns which processes:

$ ps xawf -eo pid,user,cgroup,args 
  PID USER     CGROUP                      COMMAND
 1338 root     name=systemd:/user/carla/2       \_ gdm-session-worker [pam/gdm-password]
 1358 carla    name=systemd:/user/carla/2           \_ /bin/sh /etc/xdg/xfce4/xinitrc
 1487 carla    name=systemd:/user/carla/2               \_ /usr/bin/ssh-agent /bin/sh -c exec -l /bin/bash -c "startxfce4"
 1515 carla    name=systemd:/user/carla/2               \_ xscreensaver -no-splash
 1517 carla    name=systemd:/user/carla/2               \_ xfce4-session

cgroups were introduced into the Linux kernel a few years ago, and they are an interesting mechanism for allocating and limiting kernel resources. In systemd, cgroups are used to corral and manage processes. When new processes are spawned they become members of the parent's cgroup. The cgroup is named for the service it belongs to, and services cannot escape from their cgroups so you always know what service they belong to. When you need to kill a service you can kill the cgroup, and snag all of its processes in one swoop instead of playing find-the-fork-or-renamed-process. Another way to view the process hierarchy is with the system-cgls command, as shown in Figure 2.

Figure 2: Partial output from system-cgls showing process cgroups.

There is my old friend Avahi daemon. So instead of hunting down and killing the two Avahi processes the old-fashioned way, systemd lets me do it in one command:

# systemctl kill avahi-daemon.service

Lennart Poettering, the main author of systemd, has written a series of articles for sysadmins. They're not indexed, so here are links for your convenience. These cover customizing startup services, runlevels, gettys, and everything you need to know to control systemd

•         Rethinking PID 1
• systemd for Administrators, Part 1
• systemd for Administrators, Part II
• systemd for Administrators, Part III
• systemd for Administrators, Part IV
• systemd for Administrators, Part V
• systemd for Administrators, Part VI
• systemd for Administrators, Part VII
• systemd for Administrators, Part VIII
• systemd for Administrators, Part IX
• systemd for Administrators, Part X
• systemd for Administrators, Part XI

How to Craft a Killer Cover Letter for Linux Jobs

You're certified, bona fide, and active in your favorite open source project, but how do you craft the clever cover letter that lands your next Linux job?

If you're lucky, your reputation precedes you and dream job offers land in your in-box every other day. If you're like the rest of us, a well-crafted cover letter can make the difference between getting a phone call, or getting shuffled to the bottom of the resume pile. Before prospective employers look down the list of Linux skills and open source project experience you've laid out on your resume, they'll do a quick read of your cover letter, which is your one chance to make a fabulous first impression.

I've long hated writing cover letters. Once you've written your resume, it's pretty much good to go, with occasional updates or tweaks to highlight skills that would appeal to specific employers. Cover letters, on the other hand, need to be written with a specific employer or position in mind. And knowing that these few paragraphs need to say more than your resume does can be nerve wracking. With a few rules in mind, you can take the torture out of writing and focus on why you are the only logical solution to an employer's high-tech needs.

1. Be Concise

First, the good news: No one wants to read your dissertation. Instead, keep your cover letter brief, yet meaty (like this rule).

2. Be Relevant

Before writing anything, research the company and position for which you are applying. Read the press releases on the official company site, search the web to learn more about what others are saying about the company, and consider how your skill set will be an asset to the employer. Remember that the employer wants to know how you will meet his or her needs, so don't focus on why this job would meet your needs (unless your needs are all about focusing on the needs of your prospective employer).

In her Dice.com article called How to Target A Cover to Get the Manager’s Attention, Leslie Stevens-Huffman explains, "Don't belabor points or regurgitate the information in your resume. Create a short, compelling narrative that proves you understand the company’s needs and describes how you intend to meet them."

Gayle Laakmann McDowell, author of The Google Resume and Cracking the Coding Interview, says that how you discuss your open source work depends on the type of job for which you are applying. "If you're applying for a coding role, you should focus on the code you wrote (that is, the features)," she says. "If you're applying for a Program Management job, you should focus on the leadership aspects."

Instead of going into detail in the cover letter, you could put this open source experience under a Projects section of your resume or under Work Experience, particularly if you've spent a substantial amount of time on an open source project. "Additionally, providing a link to your GitHub page or another page where a resume screener can learn more about your coding experience is always useful," she adds.

"As a hiring manager for software engineers, I'm always happy to see that a candidate is a contributor to open source projects," says Jenson Crawford, Director of Engineering at Fetch Technologies. "Participation in open source projects tells me that technology is a passion for the candidate and it's not just a job. It also shows that the candidate is interested in making things better than they were found," he says.

Crawford agrees with McDowell when it comes to including open source experience on a cover letter. "If a candidate's open source contributions can demonstrate that the candidate has the needed qualifications or experience, include the contributions to make the connection to the hiring company's needs," he explains. "If there isn't a direct connection between the candidate's open source contributions and the qualifications listed in the job posting, or if the candidate is applying to a company without a specific job posting, include information about the contributions in a general way. Perhaps something like: I'm passionate about technology and contribute the open source projects X, Y, Z."

Let's say you are applying to be a web developer and you're interviewing for a company that requires Drupal experience. Here's your opportunity to show that you've researched the company and have the exact skills they seek. The job description says:

Experience with and a high degree of competency in Drupal is a must, including development of modules and themes, and familiarity with the Drupal API, hook system, form API, etc.

So your response to this job requirement might be something like, "In addition to developing more than two dozen Drupal modules and themes for my current employer, I'm also active in the Drupal community and recently gave a DrupalCon Lightning talk about submitting patches." Now the prospective employer knows that your resume includes required skills, but you've also added something more personal about your specific skills, which brings us to the next rule.

3. Be Professional, but Personal

Showing your personality in a resume is no easy feat, so be sure to do it in your cover letter. "Not only do you want to show that you're a good fit for the position, but you also want the reader to like you," explains Kim Isaacs, Monster Resume Expert. "Appropriate use of humor, combined with a friendly and professional tone, can help endear you to the hiring manager." Still, keep in mind that the cover letter is a formal document and not an email to your BFF, so keep the tone professional, too.

4. Be Proofed and Polished

If you're sloppy on your cover letter, employers can safely assume you'll be sloppy in your code or work habits. In addition to spell checking your letter, consider having a friend or colleague proofread it, too. Reread the job description and your letter — have you shown that you understand the position for which you are applying and that you're the ideal candidate for the role?

You have as long as you need to proof the cover letter before sending it in, so be sure that it doesn't make a trip to the circular file because it's sloppy.

If you're not quite ready to apply for that great Linux job, consider ways to get involved in the community atevents or with Linux training opportunities.

What other advice do you have for crafting clever cover letters? Share your success stories (or horror stories) in the comments below

============================Rikki Endsley================