We construct a variant of Floer homology groups and prove a gluing formula for a variant of Donaldson invariants. As a corollary, the variant of Donaldson invariants is non-trivial for connected sums of 4-manifolds which satisfy a condition for Donaldson invariants. We also show a non-existence result of compact, spin 4-manifolds with boundary some homology 3-spheres and with certain intersection forms.