Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trap boto exception InvalidSpotInstanceRequestID.NotFound when allow_missing is True #1022

Merged
merged 1 commit into from Oct 26, 2018
Merged

Trap boto exception InvalidSpotInstanceRequestID.NotFound when allow_missing is True #1022

merged 1 commit into from Oct 26, 2018

Conversation

amemni
Copy link
Contributor

@amemni amemni commented Oct 11, 2018

The following behavior occurs usually when trying to destroy an EC2 one-time spot instance whereas the associated spot instance request resource does not exist no more.

machine...> cancelling spot instance request ‘sir-w5trvcnj’... Traceback (most recent call last):
...
  File "/nix/store/8z6r39v7lh6c0iw9vs6m9xaw64nn16fw-nixops-1.6.1pre2619_da33620/lib/python2.7/site-packages/nixops/backends/ec2.py", line 282, in _get_spot_instance_request_by_id
    result = self._conn.get_all_spot_instance_requests([request_id])
  File "/nix/store/a65hivsry3s50gp37p43g1s1w4yw5rvm-python2.7-boto-2.47.0/lib/python2.7/site-packages/boto/ec2/connection.py", line 1344, in get_all_spot_instance_requests
    [('item', SpotInstanceRequest)], verb='POST')
  File "/nix/store/a65hivsry3s50gp37p43g1s1w4yw5rvm-python2.7-boto-2.47.0/lib/python2.7/site-packages/boto/connection.py", line 1186, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidSpotInstanceRequestID.NotFound</Code><Message>The spot instance request ID 'sir-w5trvcnj' does not exist</Message></Error></Errors><RequestID>0e708557-a8fc-4a7c-8d6b-e82af7a4e503</RequestID></Response>

According to documentation, also as tested, once a spot instance request is cancelled and the instance gets terminated with status code instance-terminated-no-capacity, then the spot instance request goes to state closed and the resource is gone after a couple of hours: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-bid-status.html#spot-instances-bid-status-lifecycle

Logically, the _get_spot_instance_request_by_id() function is supposed to account for such a case and raise an exception with the message "Spot instance request xxx disappeared!" when argument allow_missing = False, the above exception message is instead being raised by the boto function get_all_spot_instance_requests() causing nixops to exit immediately.

    def _get_spot_instance_request_by_id(self, request_id, allow_missing=False):
        """Get spot instance request object by id."""
        self.connect()
        result = self._conn.get_all_spot_instance_requests([request_id])
        if len(result) == 0:
            if allow_missing:
                return None
            raise EC2InstanceDisappeared("Spot instance request ‘{0}’ disappeared!".format(request_id))
        return result[0]

This change intends to trap boto exceptions with the message "InvalidSpotInstanceRequestID.NotFound" and leave the exception handling at the level of parent function get_all_spot_instance_requests(), or return None if allow_missing = True.

@amemni
Add a try except for boto exception with message InvalidSpotInstanceR…
242933b 
…equestID.NotFound when allow_missing is set to True inside function _get_spot_instance_request_by_id
@amemni amemni changed the title Trap boto exception InvalidSpotInstanceRequestID.NotFound when allow_missing is True in function get_all_spot_instance_requests() Trap boto exception InvalidSpotInstanceRequestID.NotFound when allow_missing is True Oct 11, 2018
@amemni
Copy link
Contributor Author

amemni commented Oct 11, 2018

-- cc @Chakerbh @AmineChikhaoui I believe this should be a fix for case Nixops fails to destroy a terminated spot instance when the request is already closed.

@AmineChikhaoui AmineChikhaoui merged commit 7450f2a into NixOS:master Oct 26, 2018
@amemni amemni deleted the trap-boto-missing-spot-requests branch January 24, 2019 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backends/ec2
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@amemni @AmineChikhaoui