Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand gitfs for pillar #3745

Closed
jesusaurus opened this issue Feb 15, 2013 · 19 comments
Closed

Expand gitfs for pillar #3745

jesusaurus opened this issue Feb 15, 2013 · 19 comments
Assignees
Labels
Feature new functionality including changes to functionality and code refactors, etc.
Milestone

Comments

@jesusaurus
Copy link
Contributor

I really like gitfs, and would like to see it expanded. For starters, it should be documented; but more importantly I would also like to be able to back pillar data with gitfs. I see this restructuring the config format. Maybe something like:

fileserver_backend:
  - git
pillar_backend:
  - git

gitfs:
  file:
    remotes:
      - git@server:salt-state.git
  pillar:
    remotes:
      - git@server:salt-pillar.git

Thoughts?

@thatch45
Copy link
Member

Yes, I decided to wait on this when I concluded that git-python was not the right way to go. As for pillar, this should be done in an external pillar module once we get the backend moved to libgit2.

@ghost ghost assigned basepi Feb 15, 2013
@cellofellow
Copy link

Just a thought for when this gets going, but it'd be beneficial at times to have the fileserver and pillar in the same git repo and the same branch, only in subdirectories. Would it possible to offer somewhat richer ways select where the data is on the remote, besides just url? Also specify branch/tag and path?

@lukebarton
Copy link

I agree with @cellofellow

I may be missing the philosophy behind the separation, but would it be make sense to move to something like this?:

/srv/salt/state
/srv/salt/pillar

@thatch45
Copy link
Member

From a gitfs perspective, this is viable, the thing to keep in mind is just how different pillar is from the file server, pillar generates data and the file server serves files, just because they use the same logical construction for managing sls files does not mean that they are the same.
From a gitfs perspective we would need to add the ability to only serve files from a subdirectory

@jesusaurus
Copy link
Contributor Author

Personally, I would never put the pillar and the state tree in the same repo. I want to control access to the pillar data while openly sharing the state tree.

@thatch45
Copy link
Member

@jesusaurus says it very well, pillar is meant to be isolated.

@davidblewett
Copy link

"Yes, I decided to wait on this when I concluded that git-python was not the right way to go."

Have you considered dulwich: https://github.com/jelmer/dulwich ?

@basepi
Copy link
Contributor

basepi commented May 30, 2013

Hrm, haven't seen that one, I'll have to look into it, see how mature it is.

@avnik
Copy link

avnik commented Jun 2, 2013

What wrong with git-python (if it https://github.com/gitpython-developers/GitPython )

@basepi
Copy link
Contributor

basepi commented Jun 3, 2013

That is what we are using right now. I need to powwow with @thatch45 to figure out exactly why we don't like GitPython -- I've never gotten around to asking him. =P

@jacksontj
Copy link
Contributor

I realize this issue is a bit more general, but if you were looking for pillar through git it is now merged #5732

@thatch45
Copy link
Member

thatch45 commented Jul 1, 2013

I think that it would be safe to cherry-pick this feature into 0.16.0 before the final release @basepi

@basepi
Copy link
Contributor

basepi commented Jul 1, 2013

Done.

@basepi
Copy link
Contributor

basepi commented Jul 1, 2013

Should we close this issue, as we can now use git for pillar? Or do we still want to extend the existing gitfs as well?

@thatch45
Copy link
Member

thatch45 commented Jul 1, 2013

yep

@thatch45 thatch45 closed this as completed Jul 1, 2013
@jesusaurus
Copy link
Contributor Author

@avnik @basepi git-python has some limitations that the openstack developers have run into. They all seem to boil down an assumption that the local git tree is only being used by git-python. As soon as you start running git commands outside of git-python on the local repo, you will end up in some wonky states. To avoid this, you pretty much have to run "git-update-index" regularly (not something people usually do).

@thatch45
Copy link
Member

thatch45 commented Jul 1, 2013

Yes, so far so good just because these git repos are not called outside of git-python. But still, I look forward to switching to a libgit2 system when those libs are ready!

@jesusaurus
Copy link
Contributor Author

@thatch45 I don't think this issued should be closed just yet. I really like the work being done with both the gitfs backend and the git ext_pillar, but I want to see the two consolidated a bit more, and more importantly I think there should be more consistency in their configuration. Unfortunately, with the current codebase ext_pillar and fileserver_backend work in rather different ways, which makes it difficult to consolidate the heavy git lifting into a module e.g. salt.utils.gitfs.

@basepi
Copy link
Contributor

basepi commented Sep 16, 2013

@jesusaurus Can you open a new issue for these consolidation tasks? This issue is technically resolved, as there's a git backend for pillar. I think opening a new issue will help us to see what needs to be done at a glance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature new functionality including changes to functionality and code refactors, etc.
Projects
None yet
Development

No branches or pull requests

8 participants