Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: NixOS/nix
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 056d41949e5b
Choose a base ref
...
head repository: NixOS/nix
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 45dda96dd625
Choose a head ref
  • 17 commits
  • 17 files changed
  • 5 contributors

Commits on Feb 1, 2021

  1. --no-net -> --offline

    @domenkozar
    domenkozar committed Feb 1, 2021
    Copy the full SHA
    d0b74e2 View commit details

  2. Merge pull request #4500 from domenkozar/offline 

    --no-net -> --offline
    @edolstra
    edolstra authored Feb 1, 2021
    Copy the full SHA
    d0a04d1 View commit details

  3. Remove newline in operator table.

    @dschrempf
    dschrempf committed Feb 1, 2021
    Copy the full SHA
    fb00e7d View commit details

  4. Merge pull request #4501 from dschrempf/master 

    Remove newline in operator table.
    @edolstra
    edolstra authored Feb 1, 2021
    Copy the full SHA
    0e05cb6 View commit details

Commits on Feb 3, 2021

  1. Include note about type of catched errors in tryEval documentation 

    Reference #356.
    @sternenseemann
    sternenseemann committed Feb 3, 2021
    Copy the full SHA
    76d8bdf View commit details

  2. Merge pull request #4514 from sternenseemann/tryeval-doc 

    Include note about type of catched errors in tryEval documentation
    @edolstra
    edolstra authored Feb 3, 2021
    Copy the full SHA
    c8937ba View commit details

Commits on Feb 4, 2021

  1. Factor out the fetching of a cached value during eval

    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    9146c9d View commit details

  2. Add an evaluation caching mechanism 

    Add the caching mechanism already present for cli arguments to the
evaluation loop.

This means that an expression of the form `(builtins.getFlake
foo).bar.baz` will be cached if it can be
    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    49dacb2 View commit details

  3. Manually commit to the cache db during the eval 

    This is a dirty hack to make caching temporarily work.

The issue seems to be that the cache object is never deleted, which in
turn means that we never commit to the db (as it happens in the object's
destructor), so the cache doesn't persist between two Nix calls.
So we manually commit to the db each time. This is both slower and ugly,
but at least it works for the moment
    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    480a523 View commit details

  4. Only add an eval cache to the attribute sets 

    The others don't need it and it was taking some useless space in the
`Value` type
    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    57b34c5 View commit details

  5. make the eval cache reentrant 

    Avoids sqlite lock issue when the same flake is evaluated twice by
keeping an in-memory mapping of the db connections opened for each cache
key
    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    e739f46 View commit details

  6. Factor out a function for feching an attribute set element

    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    7202659 View commit details

  7. Fix a clang warning 

    Re-order the initialisers in a constructor to make them match the
actual initialisation order
    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    77f9a29 View commit details

  8. Use the eval cache in DrvInfo

    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    72d26f8 View commit details

  9. Cache the coertion of drvs to strings

    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    77ea6eb View commit details

  10. Fix the build of the test plugin

    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    a2aee4d View commit details

  11. Cache the evaluation of drv arguments

    @thufschmitt
    thufschmitt committed Feb 4, 2021
    Copy the full SHA
    45dda96 View commit details
3 changes: 1 addition & 2 deletions doc/manual/src/expressions/language-operators.md
View file
Original file line number Diff line number Diff line change
@@ -25,5 +25,4 @@ order of precedence (from strongest to weakest binding).
| Inequality | *e1* `!=` *e2* | none | Inequality. | 11 |
| Logical AND | *e1* `&&` *e2* | left | Logical AND. | 12 |
| Logical OR | *e1* `\|\|` *e2* | left | Logical OR. | 13 |
| Logical Implication | *e1* `->` *e2* | none | Logical implication (equivalent to `!e1 \|\|
e2`). | 14 |
| Logical Implication | *e1* `->` *e2* | none | Logical implication (equivalent to `!e1 \|\| e2`). | 14 |
29 changes: 0 additions & 29 deletions src/libcmd/installables.cc
View file
Original file line number Diff line number Diff line change
@@ -457,35 +457,6 @@ Value * InstallableFlake::getFlakeOutputs(EvalState & state, const flake::Locked
return aOutputs->value;
}

ref<eval_cache::EvalCache> openEvalCache(
EvalState & state,
std::shared_ptr<flake::LockedFlake> lockedFlake)
{
auto fingerprint = lockedFlake->getFingerprint();
return make_ref<nix::eval_cache::EvalCache>(
evalSettings.useEvalCache && evalSettings.pureEval
? std::optional { std::cref(fingerprint) }
: std::nullopt,
state,
[&state, lockedFlake]()
{
/* For testing whether the evaluation cache is
complete. */
if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0")
throw Error("not everything is cached, but evaluation is not allowed");

auto vFlake = state.allocValue();
flake::callFlake(state, *lockedFlake, *vFlake);

state.forceAttrs(*vFlake);

auto aOutputs = vFlake->attrs->get(state.symbols.create("outputs"));
assert(aOutputs);

return aOutputs->value;
});
}

static std::string showAttrPaths(const std::vector<std::string> & paths)
{
std::string s;
4 changes: 0 additions & 4 deletions src/libcmd/installables.hh
View file
Original file line number Diff line number Diff line change
@@ -130,8 +130,4 @@ struct InstallableFlake : InstallableValue
FlakeRef nixpkgsFlakeRef() const override;
};

ref<eval_cache::EvalCache> openEvalCache(
EvalState & state,
std::shared_ptr<flake::LockedFlake> lockedFlake);

}
10 changes: 10 additions & 0 deletions src/libexpr/attr-set.hh
View file
Original file line number Diff line number Diff line change
@@ -38,6 +38,16 @@ public:

private:
size_t size_, capacity_;

public:
/*
* An optional evaluation cache (for flakes in particular).
* If this is set, then trying to get a value from this attrset will first
* try to get it from the cache
*/
std::shared_ptr<eval_cache::AttrCursor> evalCache = nullptr;

private:
Attr attrs[0];

Bindings(size_t capacity) : size_(0), capacity_(capacity) { }
127 changes: 78 additions & 49 deletions src/libexpr/eval-cache.cc
View file
Original file line number Diff line number Diff line change
@@ -308,12 +308,25 @@ std::shared_ptr<AttrCursor> EvalCache::getRoot()
return std::make_shared<AttrCursor>(ref(shared_from_this()), std::nullopt);
}

void EvalCache::commit()
{
if (db) {
debug("Saving the cache");
auto state(db->_state->lock());
if (state->txn->active) {
state->txn->commit();
state->txn.reset();
state->txn = std::make_unique<SQLiteTxn>(state->db);
}
}
}

AttrCursor::AttrCursor(
ref<EvalCache> root,
Parent parent,
Value * value,
std::optional<std::pair<AttrId, AttrValue>> && cachedValue)
: root(root), parent(parent), cachedValue(std::move(cachedValue))
: parent(parent), cachedValue(std::move(cachedValue)), root(root)
{
if (value)
_value = allocRootValue(value);
@@ -374,6 +387,17 @@ std::string AttrCursor::getAttrPathStr(Symbol name) const
return concatStringsSep(".", getAttrPath(name));
}

std::optional<AttrValue> AttrCursor::getCachedValue()
{
if (root->db) {
if (!cachedValue)
cachedValue = root->db->getAttr(getKey(), root->state.symbols);
}
if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second))
return cachedValue->second;
return std::nullopt;
}

Value & AttrCursor::forceValue()
{
debug("evaluating uncached attribute %s", getAttrPathStr());
@@ -498,16 +522,13 @@ std::shared_ptr<AttrCursor> AttrCursor::findAlongAttrPath(const std::vector<Symb

std::string AttrCursor::getString()
{
if (root->db) {
if (!cachedValue)
cachedValue = root->db->getAttr(getKey(), root->state.symbols);
if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
if (auto s = std::get_if<string_t>(&cachedValue->second)) {
debug("using cached string attribute '%s'", getAttrPathStr());
return s->first;
} else
throw TypeError("'%s' is not a string", getAttrPathStr());
}
auto value = getCachedValue();
if (value) {
if (auto s = std::get_if<string_t>(&(value.value()))) {
debug("using cached string attribute '%s'", getAttrPathStr());
return s->first;
} else
throw TypeError("'%s' is not a string", getAttrPathStr());
}

auto & v = forceValue();
@@ -520,25 +541,22 @@ std::string AttrCursor::getString()

string_t AttrCursor::getStringWithContext()
{
if (root->db) {
if (!cachedValue)
cachedValue = root->db->getAttr(getKey(), root->state.symbols);
if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
if (auto s = std::get_if<string_t>(&cachedValue->second)) {
bool valid = true;
for (auto & c : s->second) {
if (!root->state.store->isValidPath(root->state.store->parseStorePath(c.first))) {
valid = false;
break;
}
}
if (valid) {
debug("using cached string attribute '%s'", getAttrPathStr());
return *s;
auto value = getCachedValue();
if (value) {
if (auto s = std::get_if<string_t>(&(value.value()))) {
bool valid = true;
for (auto & c : s->second) {
if (!root->state.store->isValidPath(root->state.store->parseStorePath(c.first))) {
valid = false;
break;
}
} else
throw TypeError("'%s' is not a string", getAttrPathStr());
}
}
if (valid) {
debug("using cached string attribute '%s'", getAttrPathStr());
return *s;
}
} else
throw TypeError("'%s' is not a string", getAttrPathStr());
}

auto & v = forceValue();
@@ -553,16 +571,13 @@ string_t AttrCursor::getStringWithContext()

bool AttrCursor::getBool()
{
if (root->db) {
if (!cachedValue)
cachedValue = root->db->getAttr(getKey(), root->state.symbols);
if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
if (auto b = std::get_if<bool>(&cachedValue->second)) {
debug("using cached Boolean attribute '%s'", getAttrPathStr());
return *b;
} else
throw TypeError("'%s' is not a Boolean", getAttrPathStr());
}
auto value = getCachedValue();
if (value) {
if (auto b = std::get_if<bool>(&(value.value()))) {
debug("using cached Boolean attribute '%s'", getAttrPathStr());
return *b;
} else
throw TypeError("'%s' is not a Boolean", getAttrPathStr());
}

auto & v = forceValue();
@@ -575,16 +590,13 @@ bool AttrCursor::getBool()

std::vector<Symbol> AttrCursor::getAttrs()
{
if (root->db) {
if (!cachedValue)
cachedValue = root->db->getAttr(getKey(), root->state.symbols);
if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
if (auto attrs = std::get_if<std::vector<Symbol>>(&cachedValue->second)) {
debug("using cached attrset attribute '%s'", getAttrPathStr());
return *attrs;
} else
throw TypeError("'%s' is not an attribute set", getAttrPathStr());
}
auto value = getCachedValue();
if (value) {
if (auto attrs = std::get_if<std::vector<Symbol>>(&(value.value()))) {
debug("using cached attrset attribute '%s'", getAttrPathStr());
return *attrs;
} else
throw TypeError("'%s' is not an attribute set", getAttrPathStr());
}

auto & v = forceValue();
@@ -627,3 +639,20 @@ StorePath AttrCursor::forceDerivation()
}

}

namespace nix {
std::shared_ptr<eval_cache::EvalCache> EvalState::openCache(Hash cacheKey, eval_cache::EvalCache::RootLoader rootLoader)
{
if (auto iter = evalCache.find(cacheKey); iter != evalCache.end())
return iter->second;

auto thisCache = std::make_shared<eval_cache::EvalCache>(
evalSettings.useEvalCache && evalSettings.pureEval
? std::optional{std::cref(cacheKey)}
: std::nullopt,
*this, rootLoader);
evalCache.insert({cacheKey, thisCache});
return thisCache;
}

}
11 changes: 9 additions & 2 deletions src/libexpr/eval-cache.hh
View file
Original file line number Diff line number Diff line change
@@ -16,11 +16,14 @@ class AttrCursor;

class EvalCache : public std::enable_shared_from_this<EvalCache>
{
public:
typedef std::function<Value *()> RootLoader;

private:
friend class AttrCursor;

std::shared_ptr<AttrDb> db;
EvalState & state;
typedef std::function<Value *()> RootLoader;
RootLoader rootLoader;
RootValue value;

@@ -34,6 +37,8 @@ public:
RootLoader rootLoader);

std::shared_ptr<AttrCursor> getRoot();

void commit();
};

enum AttrType {
@@ -68,7 +73,6 @@ class AttrCursor : public std::enable_shared_from_this<AttrCursor>
{
friend class EvalCache;

ref<EvalCache> root;
typedef std::optional<std::pair<std::shared_ptr<AttrCursor>, Symbol>> Parent;
Parent parent;
RootValue _value;
@@ -80,6 +84,7 @@ class AttrCursor : public std::enable_shared_from_this<AttrCursor>

public:

ref<EvalCache> root;
AttrCursor(
ref<EvalCache> root,
Parent parent,
@@ -114,6 +119,8 @@ public:

bool isDerivation();

std::optional<AttrValue> getCachedValue();

Value & forceValue();

/* Force creation of the .drv file in the Nix store. */
Loading