Las Safinhttps://las.rs/atom.xml2023-01-13T00:00:00ZLas Safinlas@protonmail.chhttps://las.rs/blog/all-you-need-is-hkt-s.htmlAll you need is higher kinded types2023-01-13T00:00:00ZLas Safinlas@protonmail.ch---
author: Las Safin
date: "2023-01-13"
keywords:
- higher
- kinded
- type
- hkt
- gadt
- type
- family
- computation
- haskell
title: All you need is higher kinded types
...
<div style="display: none">
```haskell
{-# LANGUAGE BlockArguments #-}
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE DerivingStrategies #-}
{-# LANGUAGE GHC2021 #-}
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE LiberalTypeSynonyms #-}
{-# LANGUAGE PatternSynonyms #-}
{-# LANGUAGE QuantifiedConstraints #-}
{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE UndecidableInstances #-}
{-# LANGUAGE UndecidableSuperClasses #-}
{-# OPTIONS_GHC -fprint-explicit-foralls -fprint-explicit-kinds #-}
{-# OPTIONS_GHC -fno-show-valid-hole-fits -fdefer-type-errors #-}
{-# OPTIONS_GHC -Wall -Wcompat -Wredundant-constraints #-}
import Data.Kind (Type)
import Data.Void (Void)
import Data.Functor.Identity (Identity (Identity))
import Unsafe.Coerce (unsafeCoerce)
```
</div>
This is a literate Haskell-file. You can open it in GHCi by doing the following:
```bash
curl -O https://las.rs/blog/all-you-need-is-hkt-s.lhs
nix develop --impure --expr \
'with import <nixpkgs> {}; mkShell {
buildInputs =
[(haskell.packages.ghc924.ghcWithPackages
(p: [ p.markdown-unlit ])
)];
}' \
-c ghci -pgmL markdown-unlit all-you-need-is-hkt-s.lhs
```
Haskellers (and even worse, dependent type brained people) are prone to making
complicated types to maximise abstraction and correctness.
However, you might be surprised to know that you (technically) need nothing more than ADTs,
HKTs, and rank-N types to do almost all of the magic you need.
**Core theorem**:
(Roughly) any valid Haskell term (or of a similar language) typed with type families and GADTs,
can be reformulated as a _semantically equivalent_ term that can be typed without type families and GADTs.
What do I mean with semantically equivalent?
When erasing types, the new term has the _same structure_,
modulo technically unnecessary wrappings using constructors
that ought to be newtype constructors.
(You could in fact aleviate this by adding more language features,
but that would ruin the point, unless there is a way of doing it
in a minimal way.)
## Type families → Data families
Type families in Haskell represent uncurriable functions that transform
one type-level term into another type-level term.
Let us consider the following trivial example:
```haskell
type family Not b where
Not 'True = 'False
Not 'False = 'True
```
It is in fact possible to transform the above into a _data family_,
and in fact turn any type family into a data family (or something equivalent
to avoid the limitation wrt. overlapping data family instances, see end of this post):
```haskell
type NotD :: Bool -> (Bool -> Type) -> Type
data family NotD b k
newtype instance NotD 'True k = NotDTrue (k 'False)
newtype instance NotD 'False k = NotDFalse (k 'True)
```
How can this be true? The intuitive and informal proof is that, at the type-level
terms of data kinds, though not types themselves, will **always be used to construct a type**.
We lose _no power_ (modulo GHC bugs and Haskell misfeatures) compared to the type family version.
In addition, since the data family uses newtype instances, the term can be kept representationally
equivalent.
Let us give an example:
```haskell
data Something (b :: Bool) where
SomethingInt :: Int -> Something 'True
SomethingDouble :: Double -> Something 'False
flipSomething :: Something b -> Something (Not b)
flipSomething (SomethingInt n) = SomethingDouble $ fromIntegral n
flipSomething (SomethingDouble n) = SomethingInt $ truncate n
```
We can transform the above into the following:
```haskell
flipSomethingD :: Something b -> NotD b Something
flipSomethingD (SomethingInt n) = NotDTrue $ SomethingDouble $ fromIntegral n
flipSomethingD (SomethingDouble n) = NotDFalse $ SomethingInt $ truncate n
```
In fact, we can even do the following:
```haskell
flipSomethingD' :: Something b -> Something (Not b)
flipSomethingD' = unsafeCoerce flipSomethingD
```
You can try running the above function and confirm it works.
Representionally, it should be equivalent to `flipSomething` modulo GHC optimisations
that might be inhibitted due to the use of <span class="unsafeCoerce">`unsafeCoerce`</span>.
## Data families → GADTs
If you look at `NotD`, you might notice that it looks awfully like a GADT.
```hs
-- :k NotD
type NotD :: Bool -> (Bool -> Type) -> Type
-- :t NotDTrue
pattern NotDTrue :: forall (k :: Bool -> Type). k 'False -> NotD 'True k
-- :t NotDFalse
pattern NotDFalse :: forall (k :: Bool -> Type). k 'True -> NotD 'False k
```
In fact, we can formulate the exact same thing, except unfortunately
it turns into a _data type_, hence unnecessary constructor tags and wrappings
are added in the representation:
```haskell
type NotG :: Bool -> (Bool -> Type) -> Type
data NotG b k where
NotGTrue :: k 'False -> NotG 'True k
NotGFalse :: k 'True -> NotG 'False k
flipSomethingG :: Something b -> NotG b Something
flipSomethingG (SomethingInt n) = NotGTrue $ SomethingDouble $ fromIntegral n
flipSomethingG (SomethingDouble n) = NotGFalse $ SomethingInt $ truncate n
```
You can **not** do the <span class="unsafeCoerce">`unsafeCoerce`</span> trick on this to turn it back into
the original `flipSomething`, given that `NotG` is (unfortunately) not a newtype.
One thing to note is that we gain slightly more power by turning it into a GADT,
notably, we can match on `NotG b k` and thus find out what `b` is.
## GADTs → ADTs
We can also go from GADTs to normal ADTs using equality proofs instead:
```haskell
type NotE :: Bool -> (Bool -> Type) -> Type
data NotE b k
= b ~ 'True => NotETrue (k 'False)
| b ~ 'False => NotEFalse (k 'True)
flipSomethingE :: Something b -> NotE b Something
flipSomethingE (SomethingInt n) = NotETrue $ SomethingDouble $ fromIntegral n
flipSomethingE (SomethingDouble n) = NotEFalse $ SomethingInt $ truncate n
```
However, equalities in constructor constraints are also somewhat exotic.
We can remove them by instead using `Data.Type.Equality.:~:`, but that
is in turn defined using a GADT, which we want to avoid.
We turn to Leibniz equality:
```haskell
type (:~:) :: forall k. k -> k -> Type
newtype (:~:) a b = EqPrf (forall p. p a -> p b)
refl :: a :~: a
refl = EqPrf id
absurd :: 'False :~: 'True -> a
absurd _ = error "absurd"
type NotEL :: Bool -> (Bool -> Type) -> Type
data NotEL b k
= NotELTrue (b :~: 'True) (k 'False)
| NotELFalse (b :~: 'False) (k 'True)
flipSomethingEL :: Something b -> NotEL b Something
flipSomethingEL (SomethingInt n) =
NotELTrue refl $ SomethingDouble $ fromIntegral n
flipSomethingEL (SomethingDouble n) =
NotELFalse refl $ SomethingInt $ truncate n
ex :: Int
ex = case flipSomethingEL (SomethingDouble 4.2) of
NotELFalse _ (SomethingInt n) -> n
NotELTrue prf _ -> absurd prf
```
## Peano arithmetic example
What is a non-trivial example then?
```haskell
data Nat = Zero | Succ Nat
data Vec (n :: Nat) (a :: Type)
= VNil (n :~: 'Zero)
| forall n'. VCons ('Succ n' :~: n) a (Vec n' a)
data Plus (x :: Nat) (y :: Nat) (k :: Nat -> Type)
= PlusZero (x :~: 'Zero) (k y)
| forall x'. PlusSucc ('Succ x' :~: x) (Plus x' ('Succ y) k)
newtype Flip (f :: a -> b -> Type) (y :: b) (x :: a)
= Flip (f x y)
twoBools :: Plus ('Succ 'Zero) ('Succ 'Zero) (Flip Vec Bool)
twoBools = PlusSucc refl $
PlusZero refl $ Flip $ VCons refl True $ VCons refl True $ VNil refl
```
Compared to the original approach where you'd simply pass in `VCons True $ VCons True $ VNil`,
what we're doing here is more akin to passing in an _execution trace_, i.e.
the execution of the "type family" is included in the program, and the type checker checks it.
### Doing existential quantification through universal quantification
Existential quantification can be expressed as universal quantification instead.
```haskell
newtype Some f = Some { runSome :: forall r. (forall a. f a -> r) -> r }
some :: f a -> Some f
some x = Some $ \r -> r x
```
Then we can instead define the above as:
```haskell
data VUCons' n a n' = VUCons' ('Succ n' :~: n) a (VecU n' a)
data VecU (n :: Nat) (a :: Type)
= VUNil (n :~: 'Zero)
| VUCons (Some (VUCons' n a))
data PlusUSucc' x y k x' = PlusUSucc' ('Succ x' :~: x) (PlusU x' ('Succ y) k)
data PlusU (x :: Nat) (y :: Nat) (k :: Nat -> Type)
= PlusUZero (x :~: 'Zero) (k y)
| PlusUSucc (Some (PlusUSucc' x y k))
twoBoolsU :: PlusU ('Succ 'Zero) ('Succ 'Zero) (Flip VecU Bool)
twoBoolsU = PlusUSucc $ some $ PlusUSucc' refl $
PlusUZero refl $ Flip $ VUCons $ some $ VUCons' refl True $
VUCons $ some $ VUCons' refl True $ VUNil refl
```
## Can we go further?
The answer is yes. I assume you are familiar with the SKI calculus, if not,
it essentially works the following way:
```haskell
s :: (a -> b -> c) -> (a -> b) -> a -> c
s = (<*>)
k :: a -> b -> a
k = pure
i :: a -> a
i = id
```
Using these three primitives, we can write any LC term
(with the same restrictions wrt. the type system used).
How can we leverage this here? We can use the CPS trick above
to define `newtype`s for the above constructs.
You could also define the AST as an ADT and then have the interpreter
be another ADT (using the Leibniz or GADT trick), but this seems simpler:
```haskell
type CPS a = (a -> Type) -> Type
infixr 0 ~>
type a ~> b = a -> CPS b
type UnCPS' :: a -> (b -> Type) -> (a ~> b) -> Type
newtype UnCPS' x k f = UnCPS' (f x k)
type UnCPS :: CPS (a ~> b) -> a ~> b
newtype UnCPS x y k = UnCPS (x (UnCPS' y k))
type S2' :: CPS (b ~> c) -> (c -> Type) -> b -> Type
newtype S2' f k x = S2' (UnCPS f x k)
type S2 :: (a ~> b ~> c) -> (a ~> b) -> a ~> c
newtype S2 f g x k = S2 (g x (S2' (f x) k))
type S1 :: (a ~> b ~> c) -> (a ~> b) ~> a ~> c
newtype S1 f g k = S1 (k (S2 f g))
type S :: (a ~> b ~> c) ~> (a ~> b) ~> a ~> c
newtype S f k = S (k (S1 f))
type (<*>) a b = S :@ a :@ b
infixl 4 <*>
type K1 :: a -> b ~> a
newtype K1 x y k = K1 (k x)
type K :: a ~> b ~> a
newtype K x k = K (k (K1 x))
type I :: a ~> a
newtype I x k = I (k x)
infixl 9 :@
type (:@) :: (a ~> c ~> d) -> a -> c ~> d
newtype (:@) x y z k = App (UnCPS (x y) z k)
```
We use the same CPS trick as used for turning type families into data families,
and thus are able to construct the following:
```hs
type S :: (a ~> b ~> c) ~> (a ~> b) ~> a ~> c
type K :: a ~> b ~> a
type I :: a ~> a
```
where `a ~> b` represents a type-level computation using CPS,
i.e. it's defined as `a -> (b -> Type) -> Type`.
Using the new application operator, `:@`, we can now move our
point~~free~~less programming to the type-level!
```haskell
type FlipS :: (a ~> b ~> c) ~> b ~> a ~> c
type FlipS = S :@ (S :@ (K :@ S) :@ (S :@ (K :@ K) :@ S)) :@ (K :@ K)
type ComposeS :: (b ~> c) ~> (a ~> b) ~> a ~> c
type ComposeS = S :@ (K :@ S) :@ K
type (.) a b = ComposeS :@ a :@ b
infixr 8 .
type TwiceS :: a ~> (a ~> a ~> b) ~> b
type TwiceS = S :@ (S :@ (K :@ S) :@ (S :@ (K :@ (S :@ I)) :@ K)) :@ K
```
What does this gain us? Essentially, type-level lambdas _without built-in language support_.
Quite powerful I'd say.
Can we then possibly go a step further? Could we, replace data kinds, with
SKI-Scott-encoded versions?
The answer is very unfortunately "no" in Haskell, because _universal quantification isn't erased_ at the type-level.
The issue:
```haskell
-- In fact Church-encoded, but point stands.
type NatS :: Type
type NatS = forall a. a ~> (a ~> a) ~> a
type ZeroS :: NatS
type ZeroS = K
-- Fails horribly!
{-
/my/file:some:place: error:
• Couldn't match kind ‘a0 -> CPS ((b0 ~> c0) ~> b0)’ with ‘Nat’
Expected kind ‘Nat ~> Nat’,
but ‘ComposeS :@ (S :@ I)’ has kind ‘(a0 ~> ((b0 ~> c0) ~> b0))
-> ((a0 ~> ((b0 ~> c0) ~> c0)) -> Type) -> Type’
• In the type ‘ComposeS :@ (S :@ I)’
In the type declaration for ‘SuccS’
|
79 | type SuccS = ComposeS :@ (S :@ I)
-}
-- type SuccS :: NatS ~> NatS
-- type SuccS = ComposeS :@ (S :@ I)
```
Though `NatS` uses a `forall`, the `forall` doesn't work as expected.
The issue becomes more clear if we reformulate it into
```hs
type SI :: forall b. ((b ~> b) ~> b) ~> (b ~> b) ~> b
type SI = S :@ I
{-
/my/file:some:place: error:
• Couldn't match kind: a0 -> CPS ((b0 ~> b0) ~> b0)
with: forall a. a ~> ((a ~> a) ~> a)
Expected kind ‘(forall a. a ~> ((a ~> a) ~> a))
~> (forall b. b ~> ((b ~> b) ~> b))’,
but ‘ComposeS :@ SI’ has kind ‘(a0 ~> ((b0 ~> b0) ~> b0))
-> ((a0 ~> ((b0 ~> b0) ~> b0)) -> Type) -> Type’
• In the type ‘ComposeS :@ SI’
In the type declaration for ‘SuccS’
|
78 | type SuccS = ComposeS :@ SI
| ^^^^^^^^^^^^^^
-}
type SuccS :: (forall a. a ~> (a ~> a) ~> a) ~> (forall b. b ~> (b ~> b) ~> b)
type SuccS = ComposeS :@ SI
```
If we take the crux of the error, it says it can't do the following
```hs
a0 ~> (b0 ~> b0) ~> b0
~
forall a. a ~> (a ~> a) ~> a
```
Both typings _would_ be valid for the above type-level term,
_if_ the `forall` had been erased. Because it isn't, and is more
akin to an implicit function, GHC can't do anything, since we don't
have real type-level lambdas, and the point is to avoid type-level lambdas.
Though we have been defeated here, we can still have some fun,
albeit I've gone back to using real GADTs to help with type (term?) inference:
```haskell
data VecImpl (n :: Nat) (a :: Type) where
VImplNil :: VecImpl 'Zero a
VImplCons :: a -> VecImpl n a -> VecImpl ('Succ n) a
-- Representation is equal to lists so we steal the instance
instance Show a => Show (VecImpl n a) where
showsPrec = unsafeCoerce $ showsPrec @[a]
show = unsafeCoerce $ show @[a]
showList = unsafeCoerce $ showList @[a]
type VecS1 :: Nat -> Type ~> Type
newtype VecS1 (n :: Nat) (a :: Type) (k :: Type -> Type)
= VecS1 (k (VecImpl n a))
type VecS :: Nat ~> Type ~> Type
newtype VecS (n :: Nat) (k :: (Type ~> Type) -> Type)
= VecS (k (VecS1 n))
type PlusImpl :: Nat -> Nat ~> Nat
data PlusImpl (x :: Nat) (y :: Nat) (k :: Nat -> Type) where
PlusImplZero :: (k y) -> PlusImpl 'Zero y k
PlusImplSucc :: PlusImpl x ('Succ y) k -> PlusImpl ('Succ x) y k
type PlusS :: Nat ~> Nat ~> Nat
newtype PlusS x k = PlusS (k (PlusImpl x))
-- We use SKI stuff to flip stuff around rather than simply using Flip as before
twoBoolsS ::
(FlipS :@ VecS :@ Bool . PlusS :@ 'Succ 'Zero)
('Succ 'Zero)
Identity
```
`twoBoolsS` is what you expect it to be, however, how do we define it?
The answer is that we ask GHC to infer it for us. Insert a typed hole, i.e. `twoBoolsS = _`, and let GHC guide you forward. Because _only one constructor_
is valid, GHC will essentially infer the term for you. When you reach
the case of `PlusImpl`, you need to try both, and one of them will make GHC err
because it's invalid.
What does the end result look like?
```haskell
twoBoolsS = App
$ UnCPS $ App $ UnCPS $ App $ UnCPS $ App $ UnCPS $ S $ UnCPS' $ S1
$ UnCPS' $ S2 $ K $ S2' $ UnCPS $ App $ UnCPS $ K $ UnCPS' $ K1 $ UnCPS'
$ S $ UnCPS' $ S1 $ UnCPS' $ S2 $ App $ UnCPS $ PlusS $ UnCPS' $ PlusImplSucc
$ PlusImplZero $ S2' $ UnCPS $ K1 $ UnCPS' $ App $ UnCPS $ App $ UnCPS
$ App $ UnCPS $ App $ UnCPS $ S $ UnCPS' $ S1 $ UnCPS' $ S2 $ App $ UnCPS
$ K $ UnCPS' $ K1 $ S2' $ UnCPS $ App $ UnCPS $ App $ UnCPS $ S $ UnCPS'
$ S1 $ UnCPS' $ S2 $ App $ UnCPS $ App $ UnCPS $ S $ UnCPS' $ S1 $ UnCPS'
$ S2 $ S $ S2' $ UnCPS $ App $ UnCPS $ K $ UnCPS' $ K1 $ UnCPS' $ K $ S2'
$ UnCPS $ App $ UnCPS $ K $ UnCPS' $ K1 $ UnCPS' $ S $ UnCPS' $ S1 $ UnCPS'
$ S2 $ K $ S2' $ UnCPS $ K1 $ UnCPS' $ S1 $ UnCPS' $ S2 $ K1 $ S2' $ UnCPS
$ VecS $ UnCPS' $ VecS1 $ Identity $ VImplCons True $ VImplCons False $ VImplNil
```
No one said this would be pr\*tty.
It took me a painstaking amount of time to do the above with GHCi.
I assume it would be faster if you used HLS. You could
theoretically also completely automate it upto the uninferrable parts, i.e.
the two booleans you choose (`True` and `False` in my case).
## Conclusion
Maybe the above wasn't a good idea after all.
Maybe there was a reason they added type families (surprisingly).
I would (unironically) however recommend using
data families instead wherever possible as they are much easier to infer
types for (given that they _are_ normal data types mostly).
I'm especially annoyed by the fact that many people create associated
type families that end in `-> Type`, yet don't use a data family.
You have absolutely no good reason most of the time!
There is one notable real-world use case however:
Given any complex data type, you can apply the above transformations to
turn it into something that you can derive `Generic` for,
if you also turn any instance of universal quantification into
explicit uses of a `Forall` type.
You can also handle recursive types by turning recursion into using
explicit fixpoints using `Fix`.
## Aside: Handling type family overlapping instances (for matching types)
Given this:
```haskell
type family IsUnit (x :: Type) :: Bool where
IsUnit () = 'True
IsUnit _ = 'False
```
How do we use the above tricks?
Converting it into a data family doesn't work:
```hs
data family IsUnitD (a :: Type) (k :: Bool -> Type)
newtype instance IsUnitD () k = IsUnitDY (k 'True)
newtype instance IsUnitD _ k = IsUnitDN (k 'False)
```
If all you want to do is improve inference, do the following instead:
```haskell
newtype IsUnitD (a :: Type) (k :: Bool -> Type) = IsUnitD (k (IsUnit a))
```
If however you want to see how to replicate this in languages with fewer features,
you could naively do:
```haskell
data IsUnitG (a :: Type) (k :: Bool -> Type) where
IsUnitGY :: (k 'True) -> IsUnitG () k
IsUnitGN :: (k 'False) -> IsUnitG a k
```
The issue is this is wrong! In the case of `()`, both constructors
are valid. We need an _inequality proof_:
```haskell
data IsUnitE (a :: Type) (k :: Bool -> Type)
= IsUnitEY (a :~: ()) (k 'True)
| IsUnitEN (a :~: () -> Void) (k 'False)
data SBool (b :: Bool) where
STrue :: SBool 'True
SFalse :: SBool 'False
handleIsUnitEY :: IsUnitE () SBool -> SBool 'True
handleIsUnitEY (IsUnitEY _ b) = b
handleIsUnitEY (IsUnitEN p _) = case p refl of
handleIsUnitEN :: IsUnitE Int SBool -> SBool 'False
handleIsUnitEN (IsUnitEY _ _) = error "impossible"
handleIsUnitEN (IsUnitEN _ b) = b
```
This is one aspect where using explicit equalities is more
powerful than using data families and GADTs.
If only the above type could be a newtype!
However, there is an issue: It is not clear to me how you would construct
`a :~: b -> Void` for two different types `a` and `b`
(i.e., the heads when in WHNF differ).
Take for example, `Int :~: ()`, which we get from the `IsUnitEY` case in
`handleIsUnitEN`: Can we somehow reduce this to `False :~: True` to make
use of the "trusted" function `absurd`? Naive solutions end up
with an open branch with `Int :~: ()` in scope again, thus resulting in a cycle,
unless you use type families etc., which ruins the point.
# About me
Type theorist.
Rolling my own crypto.
- E-mail: m<span style="display: none">dwuaidiuawhdiuh</span>e`@`{=html};la<span style="display: none">jxujxujuxjuju</span>s.rs
- GitHub: [\@L-as](https://github.com/L-as)
- Matrix: [\@Las:matrix.org](https://matrix.to/#/@Las:matrix.org)
# Posts
- [All you need is higher kinded types](/blog/all-you-need-is-hkt-s.html) - 2023-01-13
- [Using Haskell as my shell](/blog/haskell-as-shell.html) - 2021-07-23
- [vfork, close_range, and execve for launching processes in Procex](/blog/vfork_close_execve.html) - 2021-07-20
- [F2FS swap files broken and the arcane ritual to fix them](/blog/f2fs.html) - 2021-07-07
This page has a [markdown version](./all-you-need-is-hkt-s.md)
[Atom Feed](/atom.xml)
[Public PGP key (6B66 1F36 59D3 BAE7 0561 862E EA8E 9467 5140 F7F4)](/public-pgp-key.txt)
https://las.rs/blog/haskell-as-shell.htmlUsing Haskell as my shell2021-07-23T00:00:00ZLas Safinlas@protonmail.ch---
author: Las Safin
date: "2021-07-23"
keywords:
- haskell
- ghci
- shell
- procex
- guide
title: Using Haskell as my shell
...
Oddly, programmers use one programming language for their shell, yet another one to write programs.
When we need to run a lot of external commands, we use a shell scripting language,
and when we need to write algorithms, we use a "real" programming language.
The core difference can be summarized as the lack or presence
of data structures. Bash doesn't support data structures well,
something like Haskell, or any imperative language, like Python, does.
When we try to use Bash to handle structured data, it quickly goes wrong.
Take a look at [this stackoverflow answer](https://stackoverflow.com/a/45201229) for how to split a string into an array using `,` as a delimiter:
```bash
readarray -td '' a < <(awk '{ gsub(/, /,"\0"); print; }' <<<"$string, "); unset 'a[-1]';
declare -p a;
## declare -a a=([0]="Paris" [1]="France" [2]="Europe")
```
This is the answer burried underneath many other almost correct yet still incorrect answers.
Quite honestly, this is horrifying. What ought to be a simple function call,
has been mangled into something beyond recognition.
Yet, we still use Bash. For our shell, the most important thing
isn't how easy it is to split a string, it's how fast and easily you can
run an external command.
Relying on Python for that won't go well, as evident by [this stackoverflow question](https://stackoverflow.com/questions/89228/how-to-execute-a-program-or-call-a-system-command);
the clean answers depend on `sh`!
Imagine, if we could use one language for both?
There are many alternatives to Bash, but they are all fundamentally boring shells.
[Zsh](https://www.zsh.org/), [Fish](https://fishshell.com), [Oil](https://oilshell.org), [Elvish](https://elv.sh/),
[Nushell](https://www.nushell.sh/), [rc](https://doc.cat-v.org/plan_9/4th_edition/papers/rc),
[es](https://wryun.github.io/es-shell/), [XS](https://github.com/TieDyedDevil/XS), are domain specific languages,
and offer no real value as a "real" programming language.
Would you write your IRC client in Elvish?
Instead of making a shell language that can do more than Bash, why don't
we go the other way around, and try making an existing language usable as a shell?
For it to be a good shell, we want to make running external commands
as ergonomic as possible. It shouldn't require multiple lines of code
to read a file and pipe it to a process.
There are many languages with a REPL. If we were to make the Python REPL
fit for use as a shell, we could make a library to make it easier to run
external commands, but due to Python's syntax, it would likely require
a great amount of overhead to execute external processes,
what we're likely to do the most in our shell.
We could make a function with syntax similar to this:
```python
r("ls","-l")
```
While it's not a lot of *visual* noise, it's much slower to type than `ls -l`.
The `"`, `,`, and parentheses, take up our valuable time.
Haskell, on the other hand, has a much more lightweight syntax.
Throughout this blog post I manage to make the following syntax possible:
```hs
ξ#ls#_l
```
Its REPL, GHCi, is also quite featureful, importantly supporting path completion.
While longer than `ls -l`, for arguments without special characters and capital letters,
it grows at the same rate, 1 key stroke of overhead per argument.
To do this, I made my own library [Procex](https://github.com/L-as/procex).
Why did I make another library when `shh`, `shell-conduit`, `Shelly.hs`, etc. already exist?
The reason is that these solutions are all designed around [`createProcess`](https://hackage.haskell.org/package/process-1.6.12.0/docs/System-Process.html#v:createProcess).
`createProcess` doesn't support all the features you'd expect on a Unix system, notably
passing arbitrary file descriptors through to the called process.
In addition, it has the issue that launching processes takes a non-trivial amount of time,
around 0.5 seconds on my ODROID N2 (which is unusual hardware I admit).
On POSIX-like systems, you generally need to close all file descriptors
you don't want to pass on to the new process. This could be handles
to files, pipes, etc., notably, stdin, stdout, stderr are the first file descriptors
(in that order).
Depending on your system, the limit is different. On my system it's `1048576`.
`createProcess` is implemented such that it loops from `stderr+1` until this limit,
closing every file descriptor in this range.
On my ODROID N2, this takes around a second, meaning I had to wait
an extra second on every command to execute. This was not usable.
Procex doesn't have this problem, the specifics are [detailed here](vfork_close_execve.html),
though it is not of great importance to this article.
# Procex basics
Let's start off by loading Procex up in GHCi, after installing the [`procex` package](hackage.haskell.org/package/procex).
This depends on your system, but if you're using cabal it will generally be:
```bash
$ cabal update
$ cabal install procex --lib procex
$ cabal install pretty-simple --lib pretty-simple # Heavily recommended, gives us Text.Pretty.Simple.pPrint
$ ghci -Wall -Wno-type-defaults -XExtendedDefaultRules -XOverloadedStrings -interactive-print Text.Pretty.Simple.pPrint
> import Procex.Prelude
> import Procex.Shell
> import Procex.Shell.Labels
```
GHCi has a couple of problems Procex helps us work around, notably, stdin is not set
to line buffering, and changing directories doesn't affect path completion.
To fix the former, we run:
```hs
initInteractive
```
This is equivalent to `hSetBuffering stdin LineBuffering`.
The latter can be fixed by doing the following:
```hs
:set prompt-function promptFunction
```
Now we can use `cd` from [`Procex.Shell`](https://hackage.haskell.org/package/procex/docs/Procex-Shell.html),
and path completion will be different depending on your working directory,
where as before, path completion would always be from the directory you started GHCi in.
As a side effect the prompt will also be changed.
Procex has the concept of commands, which represent
a process to execute, along with the arguments and file descriptors
we want to pass to it.
To create a command, we can use the [`mq`](https://hackage.haskell.org/package/procex/docs/Procex-Quick.html#v:mq) function. After `mq`
you can write the arguments you want to pass, wrapping them in quotes,
but without any commata, parentheses or similar.
Listing the current directory:
```hs
mq "ls" "-l"
```
Or if you want to use the short syntax:
```hs
mq#ls#_l
```
The labels (prefixed with `#`) are interpreted as strings, where `_` is replaced by `-`,
since that character is illegal in labels.
The helpers you'll likely be interested in are all in [`Procex.Quick`](https://hackage.haskell.org/package/procex/docs/Procex-Quick.html).
`diff`-ing two strings, then capturing the output:
```hs
diff :: ByteString -> ByteString -> IO ByteString
diff x y = captureLazyNoThrow $ mq "diff" (pipeArgStrIn x) (pipeArgStrIn y)
```
`cat`-ing a string:
```hs
mq "cat" <<< "Hello World!\n"
```
Piping `curl` to `kak`:
```hs
mq "kak" <| mq "curl" "-sL" "ipinfo.io" -- The reverse will wait for curl to end instead of kak
```
`stat`-ing all the entries in your directory:
```hs
import System.Directory
listDirectory "." >>= mq "stat"
```
Piping `curl` to a file:
```hs
captureLazy (mq "curl" "-sL" "ipinfo.io") >>= B.writeFile "./myip.json"
```
Piping `stdout` and `stderr` to different places:
```hs
import qualified Data.ByteString.Lazy as B
mq
"nix"
"eval"
"nixpkgs#hello.name"
(pipeHOut 1 $ \_ stdout -> B.hGetContents stdout >>= B.putStr)
(pipeHOut 2 $ \_ stderr -> B.hGetContents stderr >>= B.writeFile "./log")
```
[`pipeHOut`](https://hackage.haskell.org/package/procex/docs/Procex-Process.html#v:pipeHOut) gives us the raw handle, allowing us to handle the data in Haskell, allowing us
to use all the usual Haskell libraries we'd use.
In general, it is a better idea to rely on Haskell alternatives to the tools in `coreutils`, as they are fit
for Bash and traditional shells:
- [`createDirectory`](https://hackage.haskell.org/package/directory-1.3.6.2/docs/System-Directory.html#v:createDirectory) instead of `mkdir`
- [`removeFile`](https://hackage.haskell.org/package/directory-1.3.6.2/docs/System-Directory.html#v:removeFile) instead of `rm`
- [`createSymbolicLink`](https://hackage.haskell.org/package/unix-2.7.2.2/docs/System-Posix-Files.html#v:createSymbolicLink) instead of `ln`
- [`replace-megaparsec`](https://hackage.haskell.org/package/replace-megaparsec)'s `streamEdit`, etc. instead of `sed`, `grep`, etc.
# Setting up your shell with Nix
You need to copy [this directory](https://github.com/L-as/procex/tree/master/example-shell),
fix `shellrcSrcPath`, then refer to the derivation built by `default.nix` in your
`environment.systemPackages`, or whatever you prefer.
The derivation produces a single file `bin/s` that launches your shell.
The equivalent of your `.bashrc` will be in the `ShellRC.hs` file.
GHCi commands will have to be put directly into `default.nix`.
All the imports in your `ShellRC.hs` file will in addition be available in the shell.
The `:li` command will reload the `ShellRC.hs` file from source instead
of using the pre-compiled version from the nix store.
# Setting up your shell without Nix
Let's make a `$HOME/.ghci-shell.hs` file, with the same purpose as the `.bashrc` file.
Let's for now put this inside:
```hs
:set -Wall -Wno-type-defaults -XExtendedDefaultRules -XOverloadedStrings -interactive-print Text.Pretty.Simple.pPrint
import Procex.Prelude
import Procex.Shell
import Procex.Shell.Labels
:set prompt-function promptFunction
initInteractive
```
You can then launch your shell with:
```sh
env GHCRTS="-c" ghci -ignore-dot-ghci -ghci-script "$HOME/.ghci-shell.hs"
```
This should work fine, but your init script won't be compiled, whereas it will with Nix.
# Speeding up typing
While the number of characters isn't very different compared to Bash,
there are some tricks to make it faster to type.
I'm using a [Japanese keyboard](https://en.wikipedia.org/wiki/File:Surface_type_cover_JIS_keyboard_layout_blue.jpg) with the UK layout.
I don't use the extra Japanese keys, so I have rebound the `Hiragana_Katakana`
key (2 keys right of space) to `"`, a valuable trick that is applicable to Bash too and has also
saved my fingers from unnecessary pain holding down shift.
I've also renamed `mq` to `ξ` as such:
```hs
ξ :: (QuickCmd a, ToByteString b) => b -> a
ξ = mq
```
I've bound my unused `Muhenkan` key (1 key left of space) to that to save another key stroke.
I recommend omitting extraneous spaces whenever possible, since the
code in your shell is write-once-read-never:
```hs
ξ#nix#build"nixpkgs#hello"#_o#out
```
Since I need to hold down shift to type `_`, I've mapped my unused `Henkan` key (1 key right of space) to it
to save one more key stroke.
My `.XCompose`:
```
<Henkan> : "_"
<Hiragana_Katakana> : "\""
<Muhenkan> : "ξ"
```
You're likely better off doing this by modifying your XKB layout,
but I didn't want to delve into that mess.
With this we're down to 34 key strokes on my keyboard.
The equivalent command in Bash:
```bash
nix build nixpkgs#hello -o out
```
This took me 31 key strokes, surprisingly quite close!
You could further save key strokes by renaming functions in Procex to shorter
names, however, I am of the belief that the user should choose
the names, not just for functions from Procex, but also for other common functions they use. I've myself made aliases
to [`Data.ByteString.Lazy.UTF8.toString`](https://hackage.haskell.org/package/utf8-string-1.0.2/docs/Data-ByteString-Lazy-UTF8.html#v:toString),
[`Data.ByteString.Lazy.UTF8.fromString`](https://hackage.haskell.org/package/utf8-string-1.0.2/docs/Data-ByteString-Lazy-UTF8.html#v:fromString),
and some other common functions I use a lot.
# Internal design of Procex
The first step was making my own glue code in C for interfacing
with the `vfork` and `execve` for creating processes, as [detailed here](vfork_close_execve.html).
You could do this in Haskell if you're careful, but file descriptors in the child,
which would effectively be another Haskell thread, would point to different things
than the parent. This is problematic since handles from the environment will
now suddenly point to different things, but only in the child.
Because of this the code that runs in the child before `execve` is in C.
If you didn't bother reading the above article, the gist is that
[the glue code](https://github.com/L-as/procex/blob/master/cbits/glue.c)
provides functions that combine the forking and execution, in addition to allowing
file descriptors to be set up for the child.
This is then bound to inside [`Procex.Execve`](https://github.com/L-as/procex/blob/master/Procex/Execve.hs).
We interface to it from [`Procex.Core`](https://github.com/L-as/procex/blob/master/Procex/Core.hs), which
defines the core `Cmd` type.
`Cmd` is internally `Args -> IO (Async ProcessStatus)`, where `Args` is a record
of the raw arguments to pass as `ByteString`s, the file descriptors to pass (and how to map them),
and what "executor" to use (used to allow `exec`-ing without `fork`-ing).
This design was chosen as it is easy to compose.
The exported functions are:
- `makeCmd' :: ByteString -> Cmd`: Takes the path to an executable and gives you a `Cmd`
- `passArg :: ByteString -> Cmd -> Cmd`: Passes an argument
- `passFd :: (Fd, Fd) -> Cmd -> Cmd`: Passes the second fd to the command, renaming it to the value of the first fd
- `passArgFd :: Fd -> Cmd -> Cmd`: Passes an argument that points to the fd, while passing the fd too.
This allows process substitution, since opening the path (`/proc/self/fd/$fd`) will open what's behind the file descriptor.
- `unIOCmd :: IO Cmd -> Cmd`: Embeds the IO action inside the `Cmd`, executing the IO action when the `Cmd` is run.
- `postCmd :: (Either SomeException (Async ProcessStatus) -> IO ()) -> Cmd -> Cmd`: Runs an IO action just after the process
is launched.
- `run' :: Cmd -> IO (Async ProcessStatus)`: Runs the command and gives you the handle to a thread
that's waiting for it to finish.
- `runReplace :: Cmd -> IO ()`: Replaces the current process with the process launched by the command.
Notably, `Procex.Core` does not expose any overlapping functionality,
since it's only meant to expose the core interface.
These all internally wrap the original function passed,
resulting in a new function that takes `Args`.
When we run a command, we simply pass it an empty `Args`, then
each "layer" will add what it needs to it, then finally
reaching the root function defined in `makeCmd'`, that calls
the functions defined in the glue code (bound in [`Procex.Execve`](https://github.com/L-as/procex/blob/master/Procex/Execve.hs)).
[`Procex.Process`](https://github.com/L-as/procex/blob/master/Procex/Process.hs) provides functionality that is
commonly needed when executing processes, and wraps over `Procex.Core`.
It defines a family of `pipe*` functions, which make pipes,
then pass one end of the pipe (as a file descriptor) to the process, and the other end to something else.
In principle, we need nothing more, but this is not very ergonomic to use as a shell.
Each argument we want to pass to a process needs a `cmd & passArg "myarg"`, and `passArg`
doesn't even work when you're in a shell:
Often, in our shell, we'll pass paths as arguments, but if you pass in non-ASCII
paths to `passArg` as a literal, they will get mangled. The top bit of each byte
in the string will simply be unset by the `IsString` implementation of `ByteString`,
since it's not UTF-8 aware, so it doesn't know how to encode such bytes into the `ByteString`.
To avoid this problem, we need a helper function that takes a `String` instead of a `ByteString`,
so that we don't use `ByteString`'s `IsString` instance.
In [`Procex.Quick`](https://github.com/L-as/procex/blob/master/Procex/Quick.hs) we define a `ToByteString` class,
that has a single `toByteString` member.
It has an instance for `[a]` where `a ~ Char` (defined this way to aid type defaulting), such
that we can define functions that take any `a` where `ToByteString a`.
To attain a Bash-like syntax that is more concise, a `QuickCmd` class is defined,
with `quickCmd :: QuickCmd a => Cmd -> a`.
It has three instances:
- `QuickCmd Cmd`, which uses `id` for the definition.
- `(a ~ ()) => QuickCmd (IO a)`, which uses `run` for the definition,
i.e. it synchronously waits for it to finish and throws if the
exit code is non-zero. The reason this isn't an instance for `IO ()` is
again to aid type defaulting.
- `(QuickCmdArg a, QuickCmd b) => QuickCmd (a -> b)`, this means `quickCmd cmd` can result
in another function that takes an `a` where `QuickCmdArg a` then returns a `b` where `QuickCmd b` again.
`QuickCmdArg` has all the instances you can guess, `String`, `ByteString`, etc.
We actually can't use `ToByteString` for our instances for `QuickCmdArg`, as
that would 1) require `UndecidableInstances` and 2) make type inference
not work in a lot of cases.
Wrapping it all up, we have the `mq` function that wraps `makeCmd` and `quickCmd`,
as shown in the basic examples.
There are also various operators that wrap over `Procex.Process` and call
`Data.ByteString.Lazy.hGetContents` for you, e.g. `<<<`, `|>`, `<!|`, and the `capture*` family of functions.
The `capture*` functions all wrap `captureFdsAsHandles`, which simply runs a command and provides
the handles to the specified file descriptors.
They all output a `ByteString`, which can be read lazily or strictly.
An important part of running commands is also checking for failures.
In Bash, we have `set -e`.
In Procex, `run` runs commands synchronously and waits for them to exit,
throwing if the command failed.
This obviously works fine for `capture*` functions that wait for the command to
finish, but what about when we're using lazy IO?
The answer is more lazy IO. We attach a "finalizer"
to the `ByteString` [(GitHub)](https://github.com/L-as/procex/blob/947a97b2ec33abc1fd977a73576d69af4c3206b2/Procex/Quick.hs#L157):
```hs
attachFinalizer :: IO () -> ByteString -> IO ByteString
attachFinalizer finalizer str = B.fromChunks <$> go (B.toChunks str)
where
go' :: [BS.ByteString] -> IO [BS.ByteString]
go' [] = finalizer >> pure []
go' (x : xs) = (x :) <$> go xs
go :: [BS.ByteString] -> IO [BS.ByteString]
go = unsafeInterleaveIO . go'
```
A `Data.ByteString.Lazy.ByteString` is internally isomorphic to a list of `Data.ByteString.ByteString`.
By converting it to and then from a list of such chunks, we can insert lazy IO into it,
executing the finalizer when we reach the nil case using `unsafeInterleaveIO`.
In practice this works quite well, but some times we don't want it to err, for example when we're using
`diff`. `diff` returns a non-zero exit code when the inputs differ, but we want to ignore that,
so for each lazy `capture*` function there is a `-NoThrow` version.
This could be extended to allow filtering what exit codes you want to ignore,
but this would complicate the "quick" module, and if you want more advanced
behavior, you'd likely be better off using `Procex.Core` and `Procex.Process` directly,
then passing the resulting `Cmd -> Cmd` to `mq`.
## The label trick
[`Procex.Shell.Labels`](https://github.com/L-as/procex/blob/master/Procex/Shell/Labels.hs) contains this:
```hs
{-# OPTIONS_GHC -Wno-orphans #-}
module Procex.Shell.Labels where
import Data.Functor
import Data.Proxy (Proxy (..))
import GHC.OverloadedLabels (IsLabel (..))
import GHC.TypeLits (KnownSymbol, symbolVal)
instance (a ~ String, KnownSymbol l) => IsLabel l a where
fromLabel =
symbolVal (Proxy :: Proxy l) <&> \case
'_' -> '-'
x -> x
```
Labels like `#label` when `-XOverloadedLabels` is enabled are translated into
something like `fromLabel @"label"`.
The reason, it's `IsLabel l a` where `a ~ String` instead of
`IsLabel l String`, is that with the latter, type inference
wouldn't work properly, meaning something like `mq #echo` wouldn't
type check.
With this instance, `fromLabel @"label"` will be inferred
to be of the type `String`, causing it to be evaluated
as `"label"`.
This will likely conflict with other uses of labels, so you
might not want it if you use other libraries
that use labels.
# Conclusion after 6 months of using Haskell as my shell
In the beginning it was certainly painful, it was as if I had to relearn talking.
Thankfully GHCi provides an escape hatch: `:!` allows you to shell out to `sh` easily.
In the process of switching my shell to Haskell, I also got a lot faster at writing Haskell.
Haskell is now the primary interface through which I use my computers, and it has been
very pleasant.
I no longer have to deal with regexes, since I can whip out a full parser combinator library any time.
You could likely also include a PostgreSQL library in the shell to access databases without going through
the `psql` REPL.
I've also been removing my scripts one by one completely, replacing them with simple Haskell functions
in my `ShellRC.hs`, where they can interface with structured data rather than raw bytes.
## Future work
Advanced completion like in Fish would be quite nice,
but unfortunately GHCi is a bit hard to customize due to its integration
into the GHC source code.
Perhaps a GHCi alternative external to GHC could be implemented,
or the Idris REPL could be modified instead, since it seems
more amenable to customisation.
--------
# Related articles
- [vfork, close_range, and execve for launching processes in Procex](/blog/vfork_close_execve.html) - 2021-07-20
# About me
Type theorist.
Rolling my own crypto.
- E-mail: m<span style="display: none">dwuaidiuawhdiuh</span>e`@`{=html};la<span style="display: none">jxujxujuxjuju</span>s.rs
- GitHub: [\@L-as](https://github.com/L-as)
- Matrix: [\@Las:matrix.org](https://matrix.to/#/@Las:matrix.org)
# Posts
- [All you need is higher kinded types](/blog/all-you-need-is-hkt-s.html) - 2023-01-13
- [Using Haskell as my shell](/blog/haskell-as-shell.html) - 2021-07-23
- [vfork, close_range, and execve for launching processes in Procex](/blog/vfork_close_execve.html) - 2021-07-20
- [F2FS swap files broken and the arcane ritual to fix them](/blog/f2fs.html) - 2021-07-07
This page has a [markdown version](./haskell-as-shell.md)
[Atom Feed](/atom.xml)
[Public PGP key (6B66 1F36 59D3 BAE7 0561 862E EA8E 9467 5140 F7F4)](/public-pgp-key.txt)
https://las.rs/blog/vfork_close_execve.htmlvfork, close_range, and execve for launching processes in Procex2021-07-20T00:00:00ZLas Safinlas@protonmail.ch---
author: Las Safin
date: "2021-07-20"
keywords:
- linux
- vfork
- syscall
- haskell
- guide
title: "vfork, close_range, and execve for launching processes in Procex"
...
[`procex`](https://hackage.haskell.org/package/procex) is a Haskell library for launching processes.
No other existing Haskell package offers flexible process execution, this post details the C side,
which uses `vfork`, `close_range`, and `execve`.
-----
# Forking
If you're not familiar with how launching processes works on Linux, there are two important syscalls:
- `execve`: This replaces the current process with a new process. It takes the path to the new executable, the arguments, and the environment.
- `fork`: This clones the current process, and is the fundamental way of making a new process. Once you're in the new cloned process, you can replace it with a new process using `execve`.
Instead of `fork`, however, we will be using a closely related sycall `vfork`.
`vfork` is like `fork`, except it doesn't clone the memory. It was originally
made to conserve resources, however, in our case it is actually much more practical.
Since memory is shared, the parent will not resume until the child has exited,
and since memory is shared, it is easy to communicate between the child and parent.
It also fixes a rather annoying problem with `fork`:
Since threads aren't cloned, if a thread in the parent is holding a lock when the fork happens,
that lock will still be held in the child, even though the thread no longer exists, thus it will never be released.
This also means we can't use `malloc` and any functions that might access locks in the child, because it could
possibly attempt to acquire a lock that was held across the fork and thus never released.
In the case of `vfork`, however, the threads in the parent will have access to the same memory as in the child,
so locks will eventually get released (if the program is not buggy), avoiding the above problem entirely.
Of course it would be nice if we didn't have to use `fork` at all, because it is
[a completely nonsensical syscall](https://drewdevault.com/2018/01/02/The-case-against-fork.html).
# Piping and file descriptors
There is another important part of creating a new process, and that is setting up pipes and file descriptors.
We want to like in `bash` be able to do `cmd < file | othercmd`. To do this, we need to set up the file descriptors
in our own process before calling `execve`, since the new process inherits all our file descriptors.
This is another pain point: All file descriptors you don't close will be passed on to the new process.
This means open files, open sockets, open handles to almost all kernel interfaces you can think of might
be leaked into the new process.
There are a couple of solutions on Linux to closing extra file descriptors:
- Loop from 0 to the maximum number of file descriptors you can open, i.e. `RLIMIT_NOFILE`. (What `System.Process.createProcess` from `process` does, completely inane and horribly slow.)
- Walk through the directory `/proc/self/fd/`, which contains an entry for each open file descriptor.
- Use the new `close_range` syscall from Linux 5.9.
`close_range` is clearly the best solution if you're on a recent Linux.
An important thing to keep in mind though, is that you **must not** launch a new process
with the `RLIMIT_NOFILE` soft limit set to anything other than `FD_SETSIZE` (usually 1024),
since [`select`](https://man7.org/linux/man-pages/man2/select.2.html) is hard-coded to that limit.
We must set it back to `FD_SETSIZE` before launching a new process, and that also provides a good fallback
for older kernels: If we set `RLIMIT_NOFILE` to `FD_SETSIZE` in the new process, we only
have to close file descriptors below `FD_SETSIZE`, which is much faster than
closing potentially millions of file descriptors.
# The code
Now that all these steps are in place, we can start implementing our glue code:
```c
#define _GNU_SOURCE
#include <sys/syscall.h>
#include <linux/version.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <unistd.h>
#include <sched.h>
#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,9,0)
#include <linux/close_range.h>
// glibc does not wrap close_range so we need to do it ourselves.
static int close_range(unsigned int first) {
return syscall(__NR_close_range, first, ~0U, 0);
}
#else
static int close_range(unsigned int first) {
// There could be fds above FD_SETSIZE, but this might not be a problem
// because we reset the soft fd limit to FD_SETSIZE (1024) later.
for (unsigned int i = first; i < FD_SETSIZE; i++) close(i);
return 0;
}
#endif
// This contains the current environment.
extern char **environ;
// This is like vfork_close_execve but replaces the current process.
int close_execve(
const char *path,
char *const argv[],
char *const envp[],
int fds[],
size_t fd_count
) {
// We make sure the file desciptors in the array point to what they're
// supposed to point to, since if e.g. one pointed to stdin (fd 0),
// we want it to mean the old stdin, not the new stdin.
for (size_t i = 0; i < fd_count; i++) {
if (fds[i] != -1) {
int fd = dup(fds[i]);
if (fd == -1) return -1;
fds[i] = fd;
}
}
// Rename the file descriptors as specified,
// closing the ones we don't want.
for (int i = 0; i < fd_count; i++) {
if (fds[i] != -1) {
if (dup2(fds[i], i) == -1) return -1;
} else {
if (close(i) == -1) return -1;
}
}
// We close all file descriptors that are larger than or equal to fd_count.
if (close_range(fd_count) == -1) return -1;
// Reset fd limit for compatibility with select(), see http://0pointer.net/blog/file-descriptor-limits.html.
struct rlimit rl;
if (getrlimit(RLIMIT_NOFILE, &rl) < 0) return -1;
rl.rlim_cur = FD_SETSIZE;
if (setrlimit(RLIMIT_NOFILE, &rl) < 0) return -1;
return execve(path, argv, envp != NULL ? envp : environ);
}
// Fork, close file descriptors, then execute.
pid_t vfork_close_execve(
const char *path, // The path to executable, does not look through PATH
char *const argv[], // Will be passed verbatim to execve
// Will be passed verbatim to execve if not NULL, otherwise it will be set to the current environment
char *const envp[],
// This is an array that is fd_count long of all file descriptorswe want to share.
// In the new process, the descriptors will be renamed, fd[i] will be renamed to i using dup2.
// -1 means it will be closed.
int fds[],
size_t fd_count
) {
// We mark this volatile so it isn't contained in a register.
// In the child, we can not write to the parent's registers, only its memory.
// This is used to pass the result code to the parent.
volatile int result = 0;
// We fork. If you change this to a standard fork it would break since we
// wouldn't be able to write to the parent's memory.
pid_t pid = vfork();
// vfork had an error.
if (pid == -1) {
return -1;
// We are in the child.
} else if (pid == 0) {
result = close_execve(path, argv, envp, fds, fd_count);
_exit(1);
// We are in the parent, the child must have finished by now.
} else {
// The child had an error.
if (result != 0) {
return result;
// Success! Return the new PID.
} else {
return pid;
}
}
}
```
[This code on GitHub](https://github.com/L-as/procex/blob/master/cbits/glue.c)
--------
# Related articles
- [Using Haskell as my shell](/blog/haskell-as-shell.html) - 2021-07-23
# About me
Type theorist.
Rolling my own crypto.
- E-mail: m<span style="display: none">dwuaidiuawhdiuh</span>e`@`{=html};la<span style="display: none">jxujxujuxjuju</span>s.rs
- GitHub: [\@L-as](https://github.com/L-as)
- Matrix: [\@Las:matrix.org](https://matrix.to/#/@Las:matrix.org)
# Posts
- [All you need is higher kinded types](/blog/all-you-need-is-hkt-s.html) - 2023-01-13
- [Using Haskell as my shell](/blog/haskell-as-shell.html) - 2021-07-23
- [vfork, close_range, and execve for launching processes in Procex](/blog/vfork_close_execve.html) - 2021-07-20
- [F2FS swap files broken and the arcane ritual to fix them](/blog/f2fs.html) - 2021-07-07
This page has a [markdown version](./vfork_close_execve.md)
[Atom Feed](/atom.xml)
[Public PGP key (6B66 1F36 59D3 BAE7 0561 862E EA8E 9467 5140 F7F4)](/public-pgp-key.txt)
https://las.rs/blog/f2fs.htmlF2FS swap files broken and the arcane ritual to fix them2021-07-07T00:00:00ZLas Safinlas@protonmail.ch---
author: Las Safin
date: "2021-07-07"
keywords:
- f2fs
- swap
- linux
title: F2FS swap files broken and the arcane ritual to fix them
...
[Jump to the fix](#the-fix)
A few days ago, after updating nixpkgs on my ODROID N2 to a reivision with Linux 5.13,
it began mysteriously crashing. Eventually I figured out that it happens when it starts
swapping to the swap file, and I noticed I even get an error when I enable the swap file:
```default
[...] F2FS-fs (dm-0): Swapfile does not align to section
```
What the fuck does this mean? - I thought then.
I began searching the internet, but unfortunately, very little comes up, except some mails from lkml.org:
- <https://lkml.org/lkml/2021/5/26/333>
- <https://lore.kernel.org/lkml/YJtS3qEDFIzqc5Ki@google.com/>
In the second link, something that *seems* like a solution is described:
> Subject: Re: [PATCH v2] f2fs: avoid swapon failure by giving a warning first
> ...
> The final solution can be migrating blocks to form a section-aligned file
> internally. Meanwhile, let's ask users to do that when preparing the swap
> file initially like:
>
> ```default
> 1) create()
> 2) ioctl(F2FS_IOC_SET_PIN_FILE)
> 3) fallocate()
> ```
The message isn't very understandable, but thankfully we have a clue: `F2FS_IOC_SET_PIN_FILE`.
Looking at the [source code for F2FS](https://github.com/torvalds/linux/blob/v5.13/fs/f2fs/file.c#L4163), and finding
[this](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2192134.html), we finally have a real
explanation of what it does:
> Yes, this is persistent. `F2FS_IOC_SET_PIN_FILE` ioctl is to prevent
> file data from moving and being garbage collected, and further update
> to the file will be handled in in-place update manner.
> I don't see any document on this, but you can find the below in
> Documentation/filesystems/f2fs.rst
>
> However, once F2FS receives `ioctl(fd, F2FS_IOC_SET_PIN_FILE)` in prior to
> `fallocate(fd, DEFAULT_MODE)`, it allocates on-disk blocks addresses having
> zero or random data, which is useful to the below scenario where:
>
> 1. `create(fd)`
> 2. `ioctl(fd, F2FS_IOC_SET_PIN_FILE)`
> 3. `fallocate(fd, 0, 0, size)`
> 4. `address = fibmap(fd, offset)`
> 5. `open(blkdev)`
> 6. `write(blkdev, address)`
F2FS is a log-structured file system, which means that writes are generally written sequentially in a log-like structure.
When the kernel uses the file as swap, it updates it in-place. By marking the file as pinned in F2FS, F2FS will make sure that it's
updated in place. This doesn't explain why it worked fine before, but let's try marking our swap file with this ioctl:
# The fix
```c
#!/usr/bin/env -S tcc -run
#define _GNU_SOURCE
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <sys/ioctl.h>
#include <linux/f2fs.h>
#include <stdint.h>
int main() {
int fd = creat("./theswapfile", S_IRUSR | S_IWUSR);
uint32_t pin = 1;
if (fd == -1) { fprintf(stderr, "Error at creat: %i\n", errno); return 1; }
int r = ioctl(fd, F2FS_IOC_SET_PIN_FILE, &pin);
if (r == -1) { fprintf(stderr, "Error at ioctl: %i\n", errno); return 1; }
uint64_t len = 8LLU * 1024LLU * 1024LLU * 1024LLU; // 8 GiB
fprintf(stderr, "len: %llu\n", len);
r = fallocate(fd, 0, 0, len);
if (r == -1) { fprintf(stderr, "Error at fallocate: %i\n", errno); return 1; }
r = close(fd);
if (r == -1) { fprintf(stderr, "Error at close: %i\n", errno); return 1; }
return 0;
}
```
Success! The `uint32_t pin` parameter for `F2FS_IOC_SET_PIN_FILE` seems to act as a boolean,
judging from [this](https://github.com/torvalds/linux/blob/v5.13/fs/f2fs/file.c#L3136).
When 0, it seems to set it to unpinned, and otherwise it seems to set it to pinned.
Unfortunately for me though, my machine still crashes occasionally, but it doesn't seem to be related to swap
anymore at least!
--------
# About me
Type theorist.
Rolling my own crypto.
- E-mail: m<span style="display: none">dwuaidiuawhdiuh</span>e`@`{=html};la<span style="display: none">jxujxujuxjuju</span>s.rs
- GitHub: [\@L-as](https://github.com/L-as)
- Matrix: [\@Las:matrix.org](https://matrix.to/#/@Las:matrix.org)
# Posts
- [All you need is higher kinded types](/blog/all-you-need-is-hkt-s.html) - 2023-01-13
- [Using Haskell as my shell](/blog/haskell-as-shell.html) - 2021-07-23
- [vfork, close_range, and execve for launching processes in Procex](/blog/vfork_close_execve.html) - 2021-07-20
- [F2FS swap files broken and the arcane ritual to fix them](/blog/f2fs.html) - 2021-07-07
This page has a [markdown version](./f2fs.md)
[Atom Feed](/atom.xml)
[Public PGP key (6B66 1F36 59D3 BAE7 0561 862E EA8E 9467 5140 F7F4)](/public-pgp-key.txt)