Key/Value backend #865

mogorman · 2018-02-04T22:58:15Z

This is just the same as #624 I will make the other hundred refrence changes needed but i just wanted to get this started.

There are quite some ideas in the issue tracker about how we could improve nixops radically.

I would like to start out with a feasible iterative plan. As far as I can see, only very trivial SQL operations are every executed against the sqlite3 database; the main use case is to use transactions to support rollback in the case of exceptions. This should not be to hard to implement ourselves in memory, in combination with a key/value server that supports locking certain keys.

Needed to faclitate multiple backends (like sqlite3/other remote database)

… it in state file implementation to __db

For now it's just the underlying DB, but it resembles something that can roll back the current transaction.

backends

mogorman · 2018-02-19T07:11:47Z

@AmineChikhaoui I believe i fixed these unit test failures with my last round of commits, they fail for me stil as I don't have aws keys but they do correctly update the attrs and get past the failure point you posted.

@tomberek Your issue is coming from nix-shell over escaping things. probably because you were in the dev-shell and then executed another deeper nix-shell. can you try installing this fork of nixops and running without the nested shells? this error does not happen to me when i used <> directly

{
  "schemaVersion": 0, 
  "deployments": {
    "8f9506b0-153e-11e8-839e-024290535936": {
      "attributes": {
        "nixExprs": "[\"<trivial.nix>\", \"<trivial-vbox.nix>\"]", 
        "name": "test", 
        "configsPath": "/nix/store/dm9a1cixp8w7mylzv6vhpwv7nj0i2kqr-nixops-machines", 
        "description": "Web server"
      }, 
      "resources": {}
    }
  }
}

AmineChikhaoui · 2018-02-19T09:46:26Z

@mogorman Thanks, so the previous errors are gone but I still see some errors during the deployment tear down in the unit tests, can't post them now because it exposes some resources IDs and AWS debug output but I'll re-run it later this week and look at the errors.

mogorman · 2018-02-23T02:46:43Z

@AmineChikhaoui did you ever get a chance to get me more info as to what is broken? or can anyone help me understand how to run the test framework better? the only errors i get when running tests are those caused by lack of aws gce etc credentials so i assume the other tests passed?

andrewchambers · 2018-03-09T05:17:07Z

Could someone explain the rationale for this change? what is the point?

mogorman · 2018-03-09T05:59:09Z

the point of providing a plugable state backend is to make it easier for people to work with nixops. In the case of the json one, it makes it much easier to commit these things to git and share them among a team, in the future I want to make a mysql/postgresql backend as well so that I can manage the state there. Others have requested a backend using s3 and simpledb. this work makes it easier to do all these things.

moretea · 2018-03-09T17:11:02Z

Having a json file is the starting point to prove that we can use any K/V store. Once that's done, adding e.g. mysql, or S3 + DynamoDB should just work fine.

gleber-da · 2018-04-26T12:13:38Z

👍 to these changes. These would help us share the workload of managing our deployments in the team.

nh2 · 2018-05-27T22:08:58Z

What's the state currently? Do you only need testers for the 3 remaining backends?

moretea · 2018-05-29T06:42:35Z

@nh2 talking to me? And which backends are you referring to?

nh2 · 2018-05-29T11:35:19Z

To whomever would like this PR in, really. I was referring to

None backend done, needs to be verified

GCE done, needs to be verified

AWS done, needs to be verified

Hetzner done, needs to be verified

in the issue description.

srghma · 2018-06-09T12:48:50Z

aws test python2 tests.py tests.functional.test_backups didn't succeed for me https://pastebin.com/ZmqNeZHW
all tests are using sqlite state file, there is no tests that use json as I understand

srghma · 2018-06-09T17:37:40Z

@mogorman I've made a pr mogorman#3

srghma · 2018-06-10T17:23:16Z

nixops/state/json_file.py

+
+    def _delete_deployment(self, deployment_uuid):
+        """NOTE: This is UNSAFE, it's guarded in nixops/deployment.py. Do not call this function except from there!"""
+        self.__db.execute("delete from Deployments where uuid = ?", (deployment_uuid,))


lol, how this line got here? its causing errors

[nix-shell:~/projects/nixops]$ python2 tests.py tests.functional.test_ec2_backups ssh-security-group> creating EC2 security group ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-ssh-security-group’... ssh-security-group> adding new rules to EC2 security group ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-ssh-security-group’... my-key-pair.......> uploading EC2 key pair ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-my-key-pair’... machine...........> creating EC2 instance (AMI ‘ami-d464cba9’, type ‘c4.large’, region ‘us-east-1’)... warning: are you sure you want to destroy EC2 machine ‘machine’? (y/N) y machine...........> destroying EC2 machine... [shutting-down] [shutting-down] [terminated] warning: are you sure you want to destroy keypair ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-my-key-pair’? (y/N) y my-key-pair.......> deleting EC2 key pair ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-my-key-pair’... ssh-security-group> deleting EC2 security group `charon-15e3710a-6cce-11e8-8e29-02424049f177-ssh-security-group' ID `sg-6bbd5620'... Exception in thread Thread-4: Traceback (most recent call last): File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 47, in destroy_deployments deployment.delete() File "/home/srghma/projects/nixops/nixops/deployment.py", line 189, in delete self._state._delete_deployment(self.uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 201, in _delete_deployment self.__db.execute("delete from Deployments where uuid = ?", (deployment_uuid,)) AttributeError: 'JsonFile' object has no attribute '_JsonFile__db' warning: not all deployments have been destroyed; some resources may still exist! E ====================================================================== ERROR: tests.functional.test_ec2_backups.test_ec2_backups('json', ['/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/logical_base.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/ec2_ebs.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/ec2_base.nix'], '') ---------------------------------------------------------------------- Traceback (most recent call last): File "/nix/store/qyqsfqjpksrvaa387yk4y5bgxnslfyx8-python2.7-nose-1.3.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/nix/store/kwq9a1bjhc3w1672khipz85qrwpxj5lv-python2.7-parameterized-0.6.1/lib/python2.7/site-packages/parameterized/parameterized.py", line 280, in <lambda> nose_func = wraps(func)(lambda *args: func(*args)) File "/home/srghma/projects/nixops/tests/functional/test_ec2_backups/__init__.py", line 30, in test_ec2_backups deployment.deploy() File "/home/srghma/projects/nixops/nixops/deployment.py", line 989, in deploy self.run_with_notify('deploy', lambda: self._deploy(**kwargs)) File "/home/srghma/projects/nixops/nixops/deployment.py", line 978, in run_with_notify f() File "/home/srghma/projects/nixops/nixops/deployment.py", line 989, in <lambda> self.run_with_notify('deploy', lambda: self._deploy(**kwargs)) File "/home/srghma/projects/nixops/nixops/deployment.py", line 922, in _deploy nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker) File "/home/srghma/projects/nixops/nixops/parallel.py", line 44, in thread_fun result_queue.put((worker_fun(t), None, t.name)) File "/home/srghma/projects/nixops/nixops/deployment.py", line 895, in worker r.create(self.definitions[r.name], check=check, allow_reboot=allow_reboot, allow_recreate=allow_recreate) File "/home/srghma/projects/nixops/nixops/backends/ec2.py", line 1012, in create self.update_tags(self.vm_id, user_tags=common_tags, check=check) File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 45, in update_tags self.update_tags_using(updater, user_tags=user_tags, check=check) File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 33, in update_tags_using tags.update(self.get_common_tags()) File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 22, in get_common_tags 'CharonStateFile': "{0}@{1}:{2}".format(getpass.getuser(), socket.gethostname(), self.depl._state.db.db_file)} AttributeError: TransactionalJsonFile instance has no attribute 'db_file' -------------------- >> begin captured logging << -------------------- boto: DEBUG: Using access key provided by client. boto: DEBUG: Using secret key provided by client.

srghma · 2018-06-10T17:31:03Z

nixops/state/json_file.py

+                self._rollback()
+            else:
+                self._commit()
+            fcntl.flock(self._lock_file, fcntl.LOCK_UN)


aren't we should remove lock file? @mogorman

srghma · 2018-06-10T17:53:05Z

nixops/resources/ec2_common.py

@@ -19,7 +19,7 @@ def _retry(self, fun, **kwargs):
    def get_common_tags(self):
        tags = {'CharonNetworkUUID': self.depl.uuid,
                'CharonMachineName': self.name,
-                'CharonStateFile': "{0}@{1}:{2}".format(getpass.getuser(), socket.gethostname(), self.depl._db.db_file)}
+                'CharonStateFile': "{0}@{1}:{2}".format(getpass.getuser(), socket.gethostname(), self.depl._state.db.db_file)}


I'm getting this error

Traceback (most recent call last): File "/nix/store/qyqsfqjpksrvaa387yk4y5bgxnslfyx8-python2.7-nose-1.3.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/nix/store/kwq9a1bjhc3w1672khipz85qrwpxj5lv-python2.7-parameterized-0.6.1/lib/python2.7/site-packages/parameterized/parameterized.py", line 280, in <lambda> nose_func = wraps(func)(lambda *args: func(*args)) File "/home/srghma/projects/nixops/tests/functional/test_ec2_backups/__init__.py", line 30, in test_ec2_backups deployment.deploy() File "/home/srghma/projects/nixops/nixops/deployment.py", line 989, in deploy self.run_with_notify('deploy', lambda: self._deploy(**kwargs)) File "/home/srghma/projects/nixops/nixops/deployment.py", line 978, in run_with_notify f() File "/home/srghma/projects/nixops/nixops/deployment.py", line 989, in <lambda> self.run_with_notify('deploy', lambda: self._deploy(**kwargs)) File "/home/srghma/projects/nixops/nixops/deployment.py", line 922, in _deploy nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker) File "/home/srghma/projects/nixops/nixops/parallel.py", line 44, in thread_fun result_queue.put((worker_fun(t), None, t.name)) File "/home/srghma/projects/nixops/nixops/deployment.py", line 895, in worker r.create(self.definitions[r.name], check=check, allow_reboot=allow_reboot, allow_recreate=allow_recreate) File "/home/srghma/projects/nixops/nixops/backends/ec2.py", line 1012, in create self.update_tags(self.vm_id, user_tags=common_tags, check=check) File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 45, in update_tags self.update_tags_using(updater, user_tags=user_tags, check=check) File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 33, in update_tags_using tags.update(self.get_common_tags()) File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 22, in get_common_tags 'CharonStateFile': "{0}@{1}:{2}".format(getpass.getuser(), socket.gethostname(), self.depl._state.db.db_file)} AttributeError: TransactionalJsonFile instance has no attribute 'db_file'

So I will rename self._db_file for json state to self.db_file

srghma · 2018-06-11T08:04:55Z

@mogorman how could one convert sqlite store file to json?

srghma · 2018-06-16T10:55:07Z

nixops/state/json_file.py

+
+    def clone_deployment(self, deployment_uuid):
+        with self.db:
+            if not uuid:


this line was causing errors

[nix-shell:~/projects/nixops]$ python2 tests.py tests.functional.test_cloning_clones -vv nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] nose.config: INFO: Excluding tests matching ['^coverage-tests\\.py$'] nose.selector: INFO: /home/srghma/projects/nixops/dev-shell is executable; skipped nose.selector: INFO: /home/srghma/projects/nixops/coverage-tests.py is executable; skipped nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] nose.config: INFO: Excluding tests matching ['^coverage-tests\\.py$'] tests.functional.test_cloning_clones.test_cloning_clones('json', ('ec2', ['/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/logical_base.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/ec2_base.nix'])) ... deployment ‘7b0585d1-7144-11e8-95a9-02421304dbb8’ destroyed ERROR ====================================================================== ERROR: tests.functional.test_cloning_clones.test_cloning_clones('json', ('ec2', ['/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/logical_base.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/ec2_base.nix'])) ---------------------------------------------------------------------- Traceback (most recent call last): File "/nix/store/qyqsfqjpksrvaa387yk4y5bgxnslfyx8-python2.7-nose-1.3.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/nix/store/kwq9a1bjhc3w1672khipz85qrwpxj5lv-python2.7-parameterized-0.6.1/lib/python2.7/site-packages/parameterized/parameterized.py", line 280, in <lambda> nose_func = wraps(func)(lambda *args: func(*args)) File "/home/srghma/projects/nixops/tests/functional/test_cloning_clones/__init__.py", line 63, in test_cloning_clones deployment2 = deployment.clone() File "/home/srghma/projects/nixops/nixops/deployment.py", line 164, in clone return self._state.clone_deployment(self.uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 208, in clone_deployment if not uuid: UnboundLocalError: local variable 'uuid' referenced before assignment ---------------------------------------------------------------------- Ran 1 test in 0.098s FAILED (errors=1)

fixed with a681561

srghma · 2018-06-16T12:16:44Z

nixops/state/json_file.py

+
+    def open_deployment(self, uuid=None):
+        """Open an existing deployment."""
+        deployment = self._find_deployment(uuid=uuid)


I was getting error here

[nix-shell:~/projects/nixops]$ rm -rfd tests/state_files && python2 tests.py tests.functional.test_query_deployments -vv nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] nose.config: INFO: Excluding tests matching ['^coverage-tests\\.py$'] nose.selector: INFO: /home/srghma/projects/nixops/dev-shell is executable; skipped nose.selector: INFO: /home/srghma/projects/nixops/coverage-tests.py is executable; skipped nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] nose.config: INFO: Excluding tests matching ['^coverage-tests\\.py$'] tests.functional.test_query_deployments.test_query_deployments('json',) ... Exception in thread Thread-9: Traceback (most recent call last): File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments deployment = state.open_deployment(uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment deployment = self._find_deployment(uuid=uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment all_deployments = self.db.read()["deployments"] File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read return json.load(f) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load **kw) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded Exception in thread Thread-10: Traceback (most recent call last): File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments deployment = state.open_deployment(uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment deployment = self._find_deployment(uuid=uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment all_deployments = self.db.read()["deployments"] File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read return json.load(f) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load **kw) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded Exception in thread Thread-4: Traceback (most recent call last): File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments deployment = state.open_deployment(uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment deployment = self._find_deployment(uuid=uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment all_deployments = self.db.read()["deployments"] File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read return json.load(f) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load **kw) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded Exception in thread Thread-8: Traceback (most recent call last): File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments deployment = state.open_deployment(uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment deployment = self._find_deployment(uuid=uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment all_deployments = self.db.read()["deployments"] File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read return json.load(f) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load **kw) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded Exception in thread Thread-5: Traceback (most recent call last): File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments deployment = state.open_deployment(uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment deployment = self._find_deployment(uuid=uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment all_deployments = self.db.read()["deployments"] File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read return json.load(f) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load **kw) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded Exception in thread Thread-7: Traceback (most recent call last): File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments deployment = state.open_deployment(uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment deployment = self._find_deployment(uuid=uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment all_deployments = self.db.read()["deployments"] File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read return json.load(f) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load **kw) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded Exception in thread Thread-6: Traceback (most recent call last): File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments deployment = state.open_deployment(uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment deployment = self._find_deployment(uuid=uuid) File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment all_deployments = self.db.read()["deployments"] File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read return json.load(f) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load **kw) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded deployment ‘a93aca6b-715e-11e8-9f15-0242f1563f79’ destroyed deployment ‘a93a4770-715e-11e8-b529-0242f1563f79’ destroyed deployment ‘a93a743d-715e-11e8-87dc-0242f1563f79’ destroyed warning: not all deployments have been destroyed; some resources may still exist! ok tests.functional.test_query_deployments.test_query_deployments('nixops',) ... deployment ‘a9634280-715e-11e8-8997-0242f1563f79’ destroyed deployment ‘a9620902-715e-11e8-b304-0242f1563f79’ destroyed deployment ‘a960bc33-715e-11e8-a0db-0242f1563f79’ destroyed deployment ‘a965e614-715e-11e8-8ca3-0242f1563f79’ destroyed deployment ‘a95cd39e-715e-11e8-8113-0242f1563f79’ destroyed deployment ‘a9672c5e-715e-11e8-92eb-0242f1563f79’ destroyed deployment ‘a95e2f8a-715e-11e8-955f-0242f1563f79’ destroyed deployment ‘a9687533-715e-11e8-988f-0242f1563f79’ destroyed deployment ‘a9648ba1-715e-11e8-ae66-0242f1563f79’ destroyed deployment ‘a95f74f8-715e-11e8-b731-0242f1563f79’ destroyed ok ---------------------------------------------------------------------- Ran 2 tests in 0.607s OK [nix-shell:~/projects/nixops]$

fix a681561

srghma · 2018-06-16T17:49:38Z

nixops/state/state_helper.py

+        self._state.set_resource_attrs(self.uuid, self.id, {key:value})
+
+    def __getitem__(self, key):
+        value = self._state.get_resource_attr(self.uuid, self.id, key)


I get an error here https://pastebin.com/N8E90uke because of the inconsistency between value type:

sqlite always returns a string (though this string may contain json, number or boolean)

json state file returns already parsed value

fix e1b9158

tomberek · 2018-11-04T19:24:34Z

What is needed for this? Do you need user reports and testing?

srghma · 2018-11-04T19:36:04Z

@tomberek hi, I need hertzner testing here #969 (let's track this issue here #1036)

domenkozar · 2020-03-26T17:33:53Z

See #1264

grahamc · 2020-03-26T19:22:16Z

Hello!

Thank you for this PR.

In the past several months, some major changes have taken place in
NixOps:

Backends have been removed, preferring a plugin-based architecture.
Here are some of them:
NixOps Core has been updated to be Python 3 only, and at the
same time, MyPy type hints have been added and are now strictly
required during CI.

This is all accumulating in to what I hope will be a NixOps 2.0
release. There is a tracking issue for that:
#1242 . It is possible that
more core changes will be made to NixOps for this release, with a
focus on simplifying NixOps core and making it easier to use and work
on.

My hope is that by adding types and more thorough automated testing,
it will be easier for contributors to make improvements, and for
contributions like this one to merge in the future.

However, because of the major changes, it has become likely that this
PR cannot merge right now as it is. The backlog of now-unmergable PRs
makes it hard to see which ones are being kept up to date.

If you would like to see this merge, please bring it up to date with
master and reopen it. If the or mypy type checking fails, please
correct any issues and then reopen it. I will be looking primarily at
open PRs whose tests are all green.

Thank you again for the work you've done here, I am sorry to be
closing it now.

Graham

moretea added 30 commits March 13, 2017 19:42

Move statefile into state.file module

7035abe

Introduce different state state schemes.

6da031f

Needed to faclitate multiple backends (like sqlite3/other remote database)

Rename statefile->state in deployment.py

8269650

Move file based deployment lock to file state implementation.

c73773f

Fix indenting

8a4e34a

Bite the bullet; remove _db references from business objects + rename…

bb056ea

… it in state file implementation to __db

move get_resources_for

6cc171a

move set_deployment_attrs

813a231

move del_deployment_attr

7b58934

move get_deployment_attr

7ed3fd3

move create_resource

0db686d

Move most logic in export() to get_all_deployment_attrs in statefile.

97e0714

Make import use an atomic transaction thingy.

584d658

For now it's just the underlying DB, but it resembles something that can roll back the current transaction.

move clone_deployment

97bda1a

move part of delete_resource

77e9b93

missed a few _'s

4d33f93

move part of delete to _delete_deployment

5649e39

use state atomic

4075901

Move part of logic of rename to _rename_resource

5f39278

oh yeah, I did rename _db to __db!

6605128

Use atomic instead of _db

5881c06

Use atomic instead of _db

de9d700

move set_resource_attrs

ff23c1c

move del_resource_attr

eb2a24a

move get_resource_attr

47b3a43

move part of export to get_all_resource_attrs

6835823

Use atomic instead of _db

3739051

oh yeah, I did rename _db to __db!

66dee1d

simple typo's

6254ccb

Need to pass in object; too much coupling to refactor right now

2d1a822

mogorman added 3 commits February 18, 2018 21:43

fix variable name issue and move subclasses to common spot among

4cea4d2

backends

work on unit tests

f477f2a

make json match the behavior of sqlite

36f2479

misnamed variable

0a836b6

srghma reviewed Jun 10, 2018

View reviewed changes

srghma mentioned this pull request Jun 14, 2018

Key value state + tests refactoring #969

Closed

22 tasks

srghma reviewed Jun 16, 2018

View reviewed changes

edolstra mentioned this pull request Nov 2, 2018

SQLite should not be a hard dependency #1036

Closed

grahamc closed this Mar 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Key/Value backend #865

Key/Value backend #865

mogorman commented Feb 4, 2018 •

edited

mogorman commented Feb 19, 2018

AmineChikhaoui commented Feb 19, 2018

mogorman commented Feb 23, 2018

andrewchambers commented Mar 9, 2018

mogorman commented Mar 9, 2018

moretea commented Mar 9, 2018

gleber-da commented Apr 26, 2018

nh2 commented May 27, 2018

moretea commented May 29, 2018

nh2 commented May 29, 2018

srghma commented Jun 9, 2018

srghma commented Jun 9, 2018

srghma Jun 10, 2018

srghma Jun 10, 2018

srghma Jun 10, 2018

srghma commented Jun 11, 2018

srghma Jun 16, 2018

srghma Jun 16, 2018

srghma Jun 16, 2018

srghma Jun 16, 2018

tomberek commented Nov 4, 2018

srghma commented Nov 4, 2018 •

edited

domenkozar commented Mar 26, 2020

grahamc commented Mar 26, 2020

Key/Value backend #865

Key/Value backend #865

Conversation

mogorman commented Feb 4, 2018 • edited

mogorman commented Feb 19, 2018

AmineChikhaoui commented Feb 19, 2018

mogorman commented Feb 23, 2018

andrewchambers commented Mar 9, 2018

mogorman commented Mar 9, 2018

moretea commented Mar 9, 2018

gleber-da commented Apr 26, 2018

nh2 commented May 27, 2018

moretea commented May 29, 2018

nh2 commented May 29, 2018

srghma commented Jun 9, 2018

srghma commented Jun 9, 2018

srghma Jun 10, 2018

Choose a reason for hiding this comment

srghma Jun 10, 2018

Choose a reason for hiding this comment

srghma Jun 10, 2018

Choose a reason for hiding this comment

srghma commented Jun 11, 2018

srghma Jun 16, 2018

Choose a reason for hiding this comment

srghma Jun 16, 2018

Choose a reason for hiding this comment

srghma Jun 16, 2018

Choose a reason for hiding this comment

srghma Jun 16, 2018

Choose a reason for hiding this comment

tomberek commented Nov 4, 2018

srghma commented Nov 4, 2018 • edited

domenkozar commented Mar 26, 2020

grahamc commented Mar 26, 2020

mogorman commented Feb 4, 2018 •

edited

srghma commented Nov 4, 2018 •

edited