Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Centraldashboard does not find configmap if namespace is not "kubeflow" #6136

Closed
DomFleischmann opened this issue Sep 13, 2021 · 1 comment
Closed

Comments

@DomFleischmann
Copy link

/kind bug

What steps did you take and what happened:
When deploying the kubeflow centraldashboard in a different namespace than kubeflow it is unable to load the centraldashboard-config configmap giving the following error:

Unable to fetch ConfigMap: {
  kind: 'Status',
  apiVersion: 'v1',
  metadata: {},
  status: 'Failure',
  message: 'configmaps "centraldashboard-config" not found',
  reason: 'NotFound',
  details: { name: 'centraldashboard-config', kind: 'configmaps' },
  code: 404
}
Unable to fetch Application information: 404 page not found

This seems to be due to the namespace being hardcoded here

What did you expect to happen:
The namespace should be configurable so that the central dashboard can be deployed in a different namespace besides kubeflow.

Anything else you would like to add:
For more details check this bug

Environment:

  • Kubeflow version: 1.3 & 1.4RC0
  • Kubernetes platform: microk8s & Charmed Kubernetes
  • Kubernetes version: 1.21
  • Ubuntu 20.04
@kubeflow-bot kubeflow-bot added this to To Do in Needs Triage Sep 13, 2021
ca-scribner added a commit to canonical/kubeflow-dashboard-operator that referenced this issue Oct 20, 2021
This notifies users if they might encounter [this bug](kubeflow/kubeflow#6136).  Can be removed when upstream is fixed
ca-scribner added a commit to canonical/kubeflow-dashboard-operator that referenced this issue Oct 22, 2021
Profile CRD is removed and instead included in the kubeflow-profiles charm (see kubeflow-profiles add-crd commit for more description)

Other misc:
* separate build-deploy and relate-to-profile integration tests
* add check to set BlockedStatus when model.name!=kubeflow (limitation due to [upstream bug](kubeflow/kubeflow#6136))
* fix status reporting to avoid __init__ overwriting existing error Status' (see below).
* fix race condition(?) in relations by bumping ops to 1.2.0.  Accessing relation data from the relation data bag was (we think?) blocked by a race condition in the ops library.  Bumping to ops==1.2.0 fixes the issue

Status reporting:
Setting ActiveStatus() in init caused the accidental resetting of still valid statuses. For example:
* `config_change` hook triggers and is handled by `main()`.  Say it then hits an `OCIImageResourceError` in `main()`, setting an `ErrorStatus()`
* `update_status` hook triggers and has no handler.  It still runs `__init__()` which passes the `except NoVersionsListed/except NoCompatibleVersions` block, which sets `ActiveStatus`.  There is no additional handling on `update_status`, so it ends with the app in `ActiveStatus`

This masks any errors triggered by other hooks, making it harder for CI and humans to notice and fix errors (this also might happen instantly if an `update_status` is queued behind some other hook.
@stale
Copy link

stale bot commented Mar 2, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the lifecycle/stale label Mar 2, 2022
@stale stale bot closed this as completed Apr 18, 2022
Needs Triage automation moved this from To Do to Closed Apr 18, 2022
@kubeflow-bot kubeflow-bot removed this from Closed in Needs Triage Apr 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants