I'm surprised to see that very little is known about the Linux kernel keyring. With keyctl[0] you can put secrets scoped to process and ensure that they stay there only for a limited period of time. The tool isn't intuitive, but it's the way I put my 2FA and secrets in shell without bothering about leaking anything.
It's definitely a powerful approach; I don't think it's particularly viable for the sort of use cases where you're throwing secrets around in a shell:
- It's not supported natively by most software (if I wanted to use it with `curl` for example, it would only be able to replace the `rbw` example since I still need to pass the secret to curl somehow);
- I don't think it's likely to gain widespread adoption, due to being a linux-specific API;
- The API itself suffers from some poor design choices, imho; it is not currently possible to set an expiry on a keyring entry without an intermediate state where the data is loaded but no expiry is set: https://lore.kernel.org/keyrings/ygar0hbrm05.fsf@localhost/T...
It's really nice as a concept and when you're developing an application where you control the entire flow of the secret data, but I don't see much practical value in it for general use cases. Exposing it as a filesystem could be a potential bridge for application support (something like `curl -H @</proc/self/keyring/@u/gitlab-authorization-header`?), though I suspect that wouldn't fly upstream because files aren't generally treated as carefully as explicit secret things. Non-enumerability (`-r` on `/proc/self/keying` and `/proc/self/keyring/*`) would help here, but I still seriously doubt that the keyring maintainers would find this to be a sane proposition :)
Lately I've been using my desktop keyring/wallet to store the secrets encrypted at rest. Then on login they get injected to my shell directly from the secure storage (unlocked at login).
I feel this is probably better than plain text, but if my machine gets popped while logged on you likely have Access to active browser sessions between MFA flows and could do more damage that way.
This article does not mention that environment variables are also visible by process in /proc/*/environ (which has restrictive permissions, but is completely visible to root).
PuTTY has added a -pwfile option for use in ssh. If not exported, this interface is likely the best for non-key batch use. It seems much superior to sshpass.
The old .netrc format can be adapted for storage (which appears popular for curl), but I prefer sqlite databases, with permissions removed for all but the owner.
> This article does not mention that environment variables are also visible by process in /proc/*/environ (which has restrictive permissions, but is completely visible to root).
What isn't visible to root? Maybe if you're willing to go down a really deep rabbit hole you can play that game, but I would generally explicitly exclude root from my threat model.
Defense in depth. Malware is software programmed to do a number of things, not all possible things (well at least until the attacker gets a shell, which is rather noisy). Scanning env vars is trivial, scanning the entire file system and traversing mount points is a bit harder, traversing all memory and guessing what’s a secret is a hell lot harder even for an interactive attacker. If you happen to include some malicious library doing dragnet mining and exfilatration of secrets, you’re more likely to dodge a bullet if you don’t have secrets in env vars than if you do.
As pointed out by evgpbfhnr, I do avoid using environment variables and justify it (though with different reasoning than yours).
Your justification is the kind of thing I mention as out-of-scope (for my purposes!) in my conclusion:
> There are also many bases that I don’t cover and routes through which sufficiently-smart malware could easily still obtain the secrets I’m working with.
/proc/$pid/environ, /proc/$pid/mem and other such vectors (ptrace, bpftrace, equivalents on other platforms) are real, but:
- they're not vectors of _accidental_ leakage like dumping the full process environment to logs or shell history are
- they rely on privileged access existing at the time that I'm handling the secret, while logs or shell history can be obtained _in the future_
- they're not the kind of thing I expect broad-spectrum malware to go rooting for: the memory of all processes is a lot of data to classify/exfiltrate, and if I were a malware author I'd fear that that would be far too resource-intensive and thus conspicuous. Browser cookie storage, password manager databases, keylogging, and the like are much easier and more valuable pickings.
> This article does not mention that environment variables are also visible by process in /proc/*/environ (which has restrictive permissions, but is completely visible to root).
He's explicitly not using export, so they won't show up there. Plain variables are not in the environment.
(it's good to bring up this file as well as getting inherited by child processes though)
I believe that unexported shell variables will be visible in /proc/*/mem, so it would be prudent to overwrite then unset them as soon as reasonably possible in their usage.
mem, yes, definitely. I'm not sure how you can protect yourself from that (or root user using ptrace or equivalent debugging tool) though...
Oh, memfd_secret?
The memory areas backing the file created with memfd_secret(2) are visible only to the processes that have ac‐
cess to the file descriptor. The memory region is removed from the kernel page tables and only the page tables
of the processes holding the file descriptor map the corresponding physical memory. (Thus, the pages in the re‐
gion can't be accessed by the kernel itself, so that, for example, pointers to the region can't be passed to
system calls.)
Before Linux 6.5, memfd_secret() was disabled by default and only available if the system administrator turned it on using "secretmem.enable=y" kernel parameter.
[...]
"To prevent potential data leaks of memory regions backed by memfd_secret() from a hybernation image, hybernation is prevented when there are active memfd_secret() users."
> But I definitely feel a lot more comfortable when secrets are never written to persistent unencrypted files, and being aware of these leakage vectors is helpful to avoid that!
It is very common for people to set environment variables for a server process from a config file that is readable by the application which is a bigger problem. At least put them a file that is only root readable (and have the process started by root).
I'm sure other languages have equivalents but I rarely see this.. for example I was about to say serde doesn't do it, but it looks like it's possible with a wrapper type? https://docs.rs/redactrs/latest/redactrs/
Anyway, this kind of tagging is good, I want more!
This is indeed a useful approach to limiting the scope of environment variables, and I try to use that rather than exporting when possible. Using files (especially "special" files like the command-substitution fd reference) is still preferable by a wide margin, and I hope that applications trend towards using files as the primary mechanism for passing in secrets.
I think single-secret files and filesystem permissions are superior between the presented options.
You don't need root to do what rootless podman does and create and work in directories that processes spawned from your normal user can't normally read using subuids. tmpfs to keep it off actual disks.
i usually use subshells and a project specific shell script to not have variables linger around in long-lived shell processes: ` ( . ./credentials && PW="$CRED_PW" ./the_thing ) ` so credentials can be retrieved via pass or whatever mechanism provides them.
>One way to avoid this is to prevent the command from being written to history. Bash has a configuration variable named HISTCONTROL, which when set to include ignorespace prevents commands prefixed with whitespace from being saved in history.
read -s in pdksh does nearly the opposite, saving the string to your history file! See https://man.openbsd.org/ksh#read pdksh is the system shell on OpenBSD, among others, and I just confirmed this is indeed what it does in OpenBSD.
EDIT: FWIW, ksh93 also behaves like pdksh (inherited ksh88 feature?), while zsh behaves like bash. read -s was added to bash 2.04 (2000) and zsh 4.1.1 (2003, committed 2002), both long after the flag was used in ksh--at least as early as the initial pdksh commit to OpenBSD in 1996.
Another trick I've used is to use named FIFOs for commands that expect there to be files rather than stdin/stdout. The command that spits the sensitive credential outputs to the FIFO and blocks.
The command that needs the sensitive credential to be input is pointed to the FIFO and reads it, and nothing is left over on disk or in the shell's history or memory.
Process substitution (the `<(echo ...)` approach I used in the post) is practically equivalent to this, creating a path that can be read by the shell and its child processes (and, at least as expanded, _only_ by them) just like a named FIFO -- but without the race condition mentioned by SoftTalker.
What you've hinted at and what I didn't mention in the post is that this is indeed a good way to avoid even having the secret ever be a shell variable. It's a bit of extra fiddling to turn just the token into the Authorization header, but it's certainly possible, something like this:
I was going to mention this too, it was a pretty common approach we used in batch files. There's a potential race condition if something else can read from the fifo after the secret is there but before the intended process consumes it, so you still need to be careful with permissions.
I'm surprised to see that very little is known about the Linux kernel keyring. With keyctl[0] you can put secrets scoped to process and ensure that they stay there only for a limited period of time. The tool isn't intuitive, but it's the way I put my 2FA and secrets in shell without bothering about leaking anything.
[0]: https://www.man7.org/linux/man-pages/man1/keyctl.1.html
It's definitely a powerful approach; I don't think it's particularly viable for the sort of use cases where you're throwing secrets around in a shell:
- It's not supported natively by most software (if I wanted to use it with `curl` for example, it would only be able to replace the `rbw` example since I still need to pass the secret to curl somehow);
- I don't think it's likely to gain widespread adoption, due to being a linux-specific API;
- The API itself suffers from some poor design choices, imho; it is not currently possible to set an expiry on a keyring entry without an intermediate state where the data is loaded but no expiry is set: https://lore.kernel.org/keyrings/ygar0hbrm05.fsf@localhost/T...
It's really nice as a concept and when you're developing an application where you control the entire flow of the secret data, but I don't see much practical value in it for general use cases. Exposing it as a filesystem could be a potential bridge for application support (something like `curl -H @</proc/self/keyring/@u/gitlab-authorization-header`?), though I suspect that wouldn't fly upstream because files aren't generally treated as carefully as explicit secret things. Non-enumerability (`-r` on `/proc/self/keying` and `/proc/self/keyring/*`) would help here, but I still seriously doubt that the keyring maintainers would find this to be a sane proposition :)
Nothing to be surprised about here. First time I'm learning of this existence.
I was also surprised that this would be surprising :)
Lately I've been using my desktop keyring/wallet to store the secrets encrypted at rest. Then on login they get injected to my shell directly from the secure storage (unlocked at login).
I feel this is probably better than plain text, but if my machine gets popped while logged on you likely have Access to active browser sessions between MFA flows and could do more damage that way.
This article does not mention that environment variables are also visible by process in /proc/*/environ (which has restrictive permissions, but is completely visible to root).
PuTTY has added a -pwfile option for use in ssh. If not exported, this interface is likely the best for non-key batch use. It seems much superior to sshpass.
The old .netrc format can be adapted for storage (which appears popular for curl), but I prefer sqlite databases, with permissions removed for all but the owner.
> This article does not mention that environment variables are also visible by process in /proc/*/environ (which has restrictive permissions, but is completely visible to root).
What isn't visible to root? Maybe if you're willing to go down a really deep rabbit hole you can play that game, but I would generally explicitly exclude root from my threat model.
Defense in depth. Malware is software programmed to do a number of things, not all possible things (well at least until the attacker gets a shell, which is rather noisy). Scanning env vars is trivial, scanning the entire file system and traversing mount points is a bit harder, traversing all memory and guessing what’s a secret is a hell lot harder even for an interactive attacker. If you happen to include some malicious library doing dragnet mining and exfilatration of secrets, you’re more likely to dodge a bullet if you don’t have secrets in env vars than if you do.
As pointed out by evgpbfhnr, I do avoid using environment variables and justify it (though with different reasoning than yours).
Your justification is the kind of thing I mention as out-of-scope (for my purposes!) in my conclusion:
> There are also many bases that I don’t cover and routes through which sufficiently-smart malware could easily still obtain the secrets I’m working with.
/proc/$pid/environ, /proc/$pid/mem and other such vectors (ptrace, bpftrace, equivalents on other platforms) are real, but:
- they're not vectors of _accidental_ leakage like dumping the full process environment to logs or shell history are
- they rely on privileged access existing at the time that I'm handling the secret, while logs or shell history can be obtained _in the future_
- they're not the kind of thing I expect broad-spectrum malware to go rooting for: the memory of all processes is a lot of data to classify/exfiltrate, and if I were a malware author I'd fear that that would be far too resource-intensive and thus conspicuous. Browser cookie storage, password manager databases, keylogging, and the like are much easier and more valuable pickings.
> This article does not mention that environment variables are also visible by process in /proc/*/environ (which has restrictive permissions, but is completely visible to root).
He's explicitly not using export, so they won't show up there. Plain variables are not in the environment.
(it's good to bring up this file as well as getting inherited by child processes though)
I think that once root is the adversary, all bets are off. The simplest being /proc/*/mem or hooking a debugger up to the process and pausing it...
I believe that unexported shell variables will be visible in /proc/*/mem, so it would be prudent to overwrite then unset them as soon as reasonably possible in their usage.
mem, yes, definitely. I'm not sure how you can protect yourself from that (or root user using ptrace or equivalent debugging tool) though...
Oh, memfd_secret?
Hm, this is interesting. What kernel version did you find this in? Im curious if this is exposed to other languages
From the man page: Linux 5.14.
Before Linux 6.5, memfd_secret() was disabled by default and only available if the system administrator turned it on using "secretmem.enable=y" kernel parameter. [...]
"To prevent potential data leaks of memory regions backed by memfd_secret() from a hybernation image, hybernation is prevented when there are active memfd_secret() users."
An alternative to the paste commands is vipe (vi + pipe?) from moreutils which opens an $EDITOR that returns the contents once saved and closed.
It helps with pasting special chars, newlines, and remote sessions without access to the local clipboard.
https://manpages.debian.org/jessie/moreutils/vipe.1> But I definitely feel a lot more comfortable when secrets are never written to persistent unencrypted files, and being aware of these leakage vectors is helpful to avoid that!
It is very common for people to set environment variables for a server process from a config file that is readable by the application which is a bigger problem. At least put them a file that is only root readable (and have the process started by root).
Is there any reason why you don't use a secrets manager like 1password with it's CLI tool? E.g.
>op read "op://foo/bar/password"
I touch on this possibility with the `rbw` example:
>`$ token=$(rbw get gitlab-access-token) # get the token from a command-line password manager`
> I’m also intrigued by the potential that type systems have for “tagging” secrets and preventing their propagation beyond where they’re needed
facet (rust) allows tagging fields as sensitive so they won't show up in logs: https://facet.rs/guide/attributes/#sensitive
I'm sure other languages have equivalents but I rarely see this.. for example I was about to say serde doesn't do it, but it looks like it's possible with a wrapper type? https://docs.rs/redactrs/latest/redactrs/
Anyway, this kind of tagging is good, I want more!
PHP has the SensitiveParameter attribute for treating fields/variables as sensitive.
https://www.php.net/manual/en/class.sensitiveparameter.php
An alternative to just exporting a variable is to prepend it to the command. This will keep it unexported for subsequent calls in current shell.
var=value some_command
This will still show up in /proc, but a lot of internal tools often rely on environment variables, so it’s kind of inevitable.
This is indeed a useful approach to limiting the scope of environment variables, and I try to use that rather than exporting when possible. Using files (especially "special" files like the command-substitution fd reference) is still preferable by a wide margin, and I hope that applications trend towards using files as the primary mechanism for passing in secrets.
I think single-secret files and filesystem permissions are superior between the presented options.
You don't need root to do what rootless podman does and create and work in directories that processes spawned from your normal user can't normally read using subuids. tmpfs to keep it off actual disks.
i usually use subshells and a project specific shell script to not have variables linger around in long-lived shell processes: ` ( . ./credentials && PW="$CRED_PW" ./the_thing ) ` so credentials can be retrieved via pass or whatever mechanism provides them.
related: https://news.ycombinator.com/item?id=43721228
>But wait – the token= command ends up in the history again
If you prepend your command with a space, it will not be added to your shell history.
As I mention in the post:
>One way to avoid this is to prevent the command from being written to history. Bash has a configuration variable named HISTCONTROL, which when set to include ignorespace prevents commands prefixed with whitespace from being saved in history.
20 years of administering Linux systems and I didn't know the read -s trick.
read -s in pdksh does nearly the opposite, saving the string to your history file! See https://man.openbsd.org/ksh#read pdksh is the system shell on OpenBSD, among others, and I just confirmed this is indeed what it does in OpenBSD.
EDIT: FWIW, ksh93 also behaves like pdksh (inherited ksh88 feature?), while zsh behaves like bash. read -s was added to bash 2.04 (2000) and zsh 4.1.1 (2003, committed 2002), both long after the flag was used in ksh--at least as early as the initial pdksh commit to OpenBSD in 1996.
As pdksh has aged into memory, OpenBSD's version is now known as oksh.
Android selected another fork, mksh, as their system shell. This is also included in Red Hat, along with ksh93.
I had read that zsh has strict emulation modes for ksh and bash. Is it possible that zsh behavior changes when those are triggered?
Yeah, I came to Linux from BSD and still have some ksh and csh muscle memory from The Before Time
Note that "read -s" is not POSIX, and thus is not portable.
The only allowed argument to POSIX read is "-r" which is enforced in Debian/Ubuntu by the dash system shell.
https://pubs.opengroup.org/onlinepubs/9699919799.2018edition...
another question here is tools that want to write their secrets to files
(looking at you `gcloud`)
best practice I've heard is to create a user fs mount that prompts every time it's accessed?
Another trick I've used is to use named FIFOs for commands that expect there to be files rather than stdin/stdout. The command that spits the sensitive credential outputs to the FIFO and blocks.
The command that needs the sensitive credential to be input is pointed to the FIFO and reads it, and nothing is left over on disk or in the shell's history or memory.
Process substitution (the `<(echo ...)` approach I used in the post) is practically equivalent to this, creating a path that can be read by the shell and its child processes (and, at least as expanded, _only_ by them) just like a named FIFO -- but without the race condition mentioned by SoftTalker.
What you've hinted at and what I didn't mention in the post is that this is indeed a good way to avoid even having the secret ever be a shell variable. It's a bit of extra fiddling to turn just the token into the Authorization header, but it's certainly possible, something like this:
curl -H @<(rbw get gitlab-access-token | sed 's/^/Authorization: Bearer/') https://gitlab.example.com/api/v4/projects
I was going to mention this too, it was a pretty common approach we used in batch files. There's a potential race condition if something else can read from the fifo after the secret is there but before the intended process consumes it, so you still need to be careful with permissions.
would very much like to see a small example of how to create, consume, and destroy those FIFOs...
https://man7.org/linux/man-pages/man1/mkfifo.1.html
Pretty simple. This creates a named pipe. One end of a shell command redirects to it, one end redirects from it. rm when finished.
You can just use process substitution
cat <(secret-print my-secret)
In a shell script situation, you'd typically trap EXIT and ERR and remove the fifo in the handler.